Add support for strings and byte arrays

[pavel-kirienko]

Implement string and byte array support as described in OpenCyphal/specification#51. The Specification will be updated after this is agreed upon. The grammar has been modified as follows:

• Add utf8 primitive.
• Add byte primitive.
• Remove the unnecessary form saturated bool, which was shorthand for bool, for regularity. From now on, only unqualified bool is allowed. The expected impact is zero since the long form was never seen in the wild. Note that truncated bool was never legal.

At the moment, the following constraints are enforced; they may be lifted in the future:

• utf8 can only be used as an element type of a variable-length array:

utf8[<=10] name  # valid
utf8[10]   name  # invalid
utf8       name  # invalid

• byte can only be used as an element type of an array:

byte[<=10] data  # valid
byte[10]   data  # valid
byte       data  # invalid

Both utf8 and byte are concrete instances of UnsignedIntegerType. By virtue of this, existing users of PyDSDL (like Nunavut) will interpret both utf8 and byte as truncated uint8, until explicitly updated to take advantage of the new feature. That is:

utf8[<=10] name  # seen by Nunavut as: truncated uint8[<=10]
utf8[10]   name  # frontend error
utf8       name  # frontend error
byte[<=10] data  # seen by Nunavut as: truncated uint8[<=10]
byte[10]   data  # seen by Nunavut as: truncated uint8[10]
byte       data  # frontend error

Property pydsdl.ArrayType.string_like is now deprecated and should be replaced with an explicit isinstance(array.element_type, pydsdl.UTF8Type).

Some internal refactoring has been done to unify the deprecation consistency checking with the aggregation constraints listed above (e.g., utf8[<=10] is valid but utf8[10] is not).

Closes #96
Relates to OpenCyphal/yakut#65

[pavel-kirienko]

I would like to summon @samcrow for commentary.

[samcrow]

This looks good overall and not very difficult to add to other implementations.

[aentinger]

May I inquire if there's anything keeping this PR from being merged?

[pavel-kirienko]

Blocked on this OpenCyphal/nunavut#301

…

On Sun, Jul 2, 2023, 20:22 Alexander Entinger ***@***.***> wrote: May I inquire if there's anything keeping this PR from being merged? — Reply to this email directly, view it on GitHub <#97 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZFIZCHRAXUNTKY5OCVGGLXOGU6FANCNFSM6AAAAAAXTPDSR4> . You are receiving this because you were assigned.Message ID: ***@***.***>

[pavel-kirienko]

@thirtytwobits I suggest we merge this

[thirtytwobits]

Given that these types are not part of the current standard, should we add an "allow experimental" or, inversely, "strict" mode to ensure we can enforce 1.0 compatibility when using pydsdl?

[pavel-kirienko]

I added a new strict mode option. In the future we should extract all options into a single Options dataclass. Right now we can't do it properly without breaking API compatibility.

[emrainey]

Would there ever be a case where we'd want to send individual characters as bytes or utf8?

Certainly one could 'byte[1]' but this seems heavy.

[pavel-kirienko]

I couldn't come up with a use case where this would be needed instead of uint8. Note that a single byte of utf8 is meaningless because it is a variable-length encoding, and the length of an encoded code point depends on the value of the first byte.

We can always remove these restrictions in the future if needed. It makes sense to start with a restricted feature set to allow more choices in the future without breaking backward compatibility.