Add support for structured dtypes to zarr3 driver, open structs as void#271
Add support for structured dtypes to zarr3 driver, open structs as void#271BrianMichell wants to merge 67 commits intogoogle:masterfrom
zarr3 driver, open structs as void#271Conversation
…t for raw bits dtype
Implement shim for `open_as_void` driver level flag
* Begin removing void field shim * Fully removed void string shim * Cleanup debug prints * Remove shimmed validation * Remove unnecessary comment * Prefer false over zero for ternary clarity
* Implement a more general and portable example set * Fix driver cache bug * Update example for template * Cleanup example * Remove testing examples from source
* Use the appropriate fill value for open_as_void structured data * Cleanup
|
I'll try to get to this in about a week, before I look this one over, please double check that the prior PR works for you. Also look over this one and see if any of the suggestions from the other one applies. |
Matches the pattern from zarr v2 driver (PR google#272). When both "field" and "open_as_void" are specified in the spec, return an error since these options are mutually exclusive - field selects a specific field from a structured array, while open_as_void provides raw byte access to the entire structure.
The zarr3 URL syntax cannot represent field selection or void access mode. Following the pattern from zarr v2 driver (PR google#272), ToUrl() now returns an error when either of these options is specified instead of silently ignoring them.
…trip Following the pattern from zarr v2 driver (PR google#272), override GetBoundSpecData in ZarrDataCache to set spec.open_as_void from ChunkCacheImpl::open_as_void_. This ensures that when you open a store with open_as_void=true and then call spec(), the resulting spec correctly has open_as_void=true set. Without this fix, opening a store with open_as_void=true and then getting its spec would lose the open_as_void flag, causing incorrect behavior if the spec is used to re-open the store.
The ZarrShardSubChunkCache template had duplicate member variables (open_as_void_, original_is_structured_, bytes_per_element_) that were already present in the base class ChunkCacheImpl (ZarrLeafChunkCache). Access these through ChunkCacheImpl:: prefix instead to follow DRY principle and maintain consistency with other TensorStore patterns.
Reviewed the code for potential inconsistencies and fixed some bugs
|
FWIW I see a new assert failure in this PR: It's best to run all the tests as you're developing. I typically do So far this is mostly build-based issues. I'll look at more of the structure later. |
…ubspan does not exceed the grid size. This prevents potential out-of-bounds access when generating keys. Resolves: google#271 (comment)
|
Ok, from an implementation perspective this is somewhat better than before. However there doesn't seem to be a spec. It does look like zarrs has some thoughts: zarrs/zarrs#154. And zarr-python has others: zarr-developers/zarr-python#2874 It would at least be reasonable to get an extension into the spec: https://zarr-specs.readthedocs.io/en/latest/v3/data-types/index.html |
|
A spec in the zarr-extensions repo would be sufficient to move forward on this --- getting it as a core data type in the zarr v3 spec would not be required (or likely to occur any time soon). However, |
…271 (comment) with TODO question pending
Supersedes #264
Resolves comments 1 and 2