Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
2be5f11
Introduce AttributeType system to replace AttributeAdapter
claude Dec 21, 2025
055c9c6
Update documentation for new AttributeType system
claude Dec 21, 2025
af9bd8d
Apply ruff-format fixes to AttributeType implementation
claude Dec 21, 2025
9bd37f6
Add DJBlobType and migration utilities for blob columns
claude Dec 21, 2025
c8d8a22
Clarify migration handles all blob type variants
claude Dec 21, 2025
61db015
Fix ruff linter errors: add migrate to __all__, remove unused import
claude Dec 21, 2025
78e0d1d
Remove serializes flag; longblob is now raw bytes
claude Dec 21, 2025
c173356
Remove unused blob imports from fetch.py and table.py
claude Dec 21, 2025
106f859
Update docs: use <djblob> for serialized data, longblob for raw bytes
claude Dec 21, 2025
e293fec
Merge branch 'claude/add-file-column-type-LtXQt' into claude/upgrade-…
dimitri-yatsenko Dec 21, 2025
d66f76e
Merge claude/add-file-column-type-LtXQt into upgrade-adapted-type
claude Dec 22, 2025
2f9b2be
Merge remote-tracking branch 'origin/claude/upgrade-adapted-type-1W3a…
claude Dec 22, 2025
1d4b7b4
Merge remote-tracking branch 'origin/pre/v2.0' into claude/upgrade-ad…
claude Dec 22, 2025
8091225
Merge remote-tracking branch 'origin/claude/add-file-column-type-LtXQ…
claude Dec 23, 2025
c34a5b8
Merge remote-tracking branch 'origin/claude/add-file-column-type-LtXQ…
claude Dec 24, 2025
cab10f6
Add storage types redesign spec
claude Dec 25, 2025
261543f
Merge remote-tracking branch 'origin/pre/v2.0' into claude/upgrade-ad…
claude Dec 25, 2025
ecac82d
Update storage types spec with OAS integration approach
claude Dec 25, 2025
7e7f968
Unify external storage under OAS with content-addressed region
claude Dec 25, 2025
495d7f7
Make <djblob@store> and <attach@store> return values transparently
claude Dec 25, 2025
7ae8f15
Introduce layered storage architecture with content core type
claude Dec 25, 2025
6fcc4d3
Add parameterized AttributeTypes and content vs object comparison
claude Dec 25, 2025
b87342b
Make content storage per-project and add migration utility
claude Dec 25, 2025
40c1dbb
Add filepath as third OAS region with ObjectRef interface
claude Dec 25, 2025
dbf092d
Redesign filepath as URI reference tracker and add json core type
claude Dec 25, 2025
43c1999
Simplify filepath to filepath@store with relative paths for portability
claude Dec 25, 2025
b9b6e34
Simplify to two-layer architecture: database types + AttributeTypes
claude Dec 25, 2025
2a5d161
Add three-layer type architecture with core DataJoint types
claude Dec 25, 2025
d36739d
Use angle brackets for all AttributeTypes in definitions
claude Dec 25, 2025
5c1e854
Add implementation plan for storage types redesign
claude Dec 25, 2025
979f45b
Implement Phase 1: Core type system with store parameter support
claude Dec 25, 2025
6926c58
Remove legacy AttributeAdapter support, update tests for AttributeType
claude Dec 25, 2025
97bc162
Simplify core type system: remove SERIALIZED_TYPES, clarify blob sema…
claude Dec 25, 2025
2de222a
Simplify type system: only core types and AttributeTypes
claude Dec 25, 2025
f35e027
Define complete core type system with blob→longblob mapping
claude Dec 25, 2025
746108a
Implement Phase 2: Content-Addressed Storage
claude Dec 25, 2025
328a59a
Apply ruff-format to content_registry.py
claude Dec 25, 2025
bbbfbc3
Remove legacy compatibility shims: attribute_adapter.py, bypass_seria…
claude Dec 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/compute/key-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ definition = """
-> Recording
---
sample_rate : float
eeg_data : longblob
eeg_data : <djblob>
"""
key_source = Recording & 'recording_type = "EEG"'
```
4 changes: 2 additions & 2 deletions docs/src/compute/make.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down Expand Up @@ -188,7 +188,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down
6 changes: 3 additions & 3 deletions docs/src/compute/populate.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ class FilteredImage(dj.Computed):
# Filtered image
-> Image
---
filtered_image : longblob
filtered_image : <djblob>
"""

def make(self, key):
Expand Down Expand Up @@ -196,7 +196,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down Expand Up @@ -230,7 +230,7 @@ class ImageAnalysis(dj.Computed):
# Complex image analysis results
-> Image
---
analysis_result : longblob
analysis_result : <djblob>
processing_time : float
"""

Expand Down
2 changes: 1 addition & 1 deletion docs/src/design/integrity.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ definition = """
-> EEGRecording
channel_idx : int
---
channel_data : longblob
channel_data : <djblob>
"""
```
![doc_1-many](../images/doc_1-many.png){: style="align:center"}
Expand Down
13 changes: 11 additions & 2 deletions docs/src/design/tables/attributes.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,10 @@ fractional digits.
Because of its well-defined precision, `decimal` values can be used in equality
comparison and be included in primary keys.

- `longblob`: arbitrary numeric array (e.g. matrix, image, structure), up to 4
- `longblob`: raw binary data, up to 4
[GiB](http://en.wikipedia.org/wiki/Gibibyte) in size.
Numeric arrays are compatible between MATLAB and Python (NumPy).
Stores and returns raw bytes without serialization.
For serialized Python objects (arrays, dicts, etc.), use `<djblob>` instead.
The `longblob` and other `blob` datatypes can be configured to store data
[externally](../../sysadmin/external-store.md) by using the `blob@store` syntax.

Expand All @@ -71,6 +72,10 @@ info).
These types abstract certain kinds of non-database data to facilitate use
together with DataJoint.

- `<djblob>`: DataJoint's native serialization format for Python objects. Supports
NumPy arrays, dicts, lists, datetime objects, and nested structures. Compatible with
MATLAB. See [custom types](customtype.md) for details.

- `object`: managed [file and folder storage](object.md) with support for direct writes
(Zarr, HDF5) and fsspec integration. Recommended for new pipelines.

Expand All @@ -80,6 +85,10 @@ sending/receiving an opaque data file to/from a DataJoint pipeline.
- `filepath@store`: a [filepath](filepath.md) used to link non-DataJoint managed files
into a DataJoint pipeline.

- `<custom_type>`: a [custom attribute type](customtype.md) that defines bidirectional
conversion between Python objects and database storage formats. Use this to store
complex data types like graphs, domain-specific objects, or custom data structures.

## Numeric type aliases

DataJoint provides convenient type aliases that map to standard MySQL numeric types.
Expand Down
Loading
Loading