Skip to content

refactor(io): serve MATLAB subscripts in the base lazy array - #945

Draft
ehennestad wants to merge 1 commit into
backend-contract/backend-probesfrom
backend-contract/lazy-array-selection
Draft

ehennestad wants to merge 1 commit into
backend-contract/backend-probesfrom
backend-contract/lazy-array-selection

Conversation

@ehennestad

Copy link
Copy Markdown
Collaborator

Each lazy array turns MATLAB subscripts (DataStub indexing, load_mat_style) into reads itself. The Zarr v3 lazy array in #853 re-implements the checks, expected-size rules and reordering of HDF5LazyArray, adds its own choice between a bounding-box read and element reads, and reverses dimensions in several places. This is the second point of the design note on #853.

io.backend.base.LazyArray now implements load_mat_style. It checks the subscripts, reads each dimension's indices once and sorted, and returns the values in the order and shape the subscripts ask for, as a table for a compound dataset. A backend implements two reads in its own storage order: readSubscripts reads every combination of per-dimension indices, and readPoints reads single elements. A linear selection is read through its bounding box or element by element, within limits the backend sets in getBoundingBoxLimits. A backend that stores arrays in row-major order returns true from isStorageOrderReversed, and the base class reverses subscripts and dimensions for it. readHyperslab serves start, count and stride through the same path.

HDF5LazyArray implements the two reads with its existing selections: one union of hyperslabs for subscripts and one element selection for points. The subscript rules and results are unchanged. A linear selection that fills its bounding box is now read as one hyperslab in any rank; before, only a contiguous run in a one-dimensional dataset was.

tests.unit.io.backend.base.LazyArraySelectionTest runs selections through an in-memory lazy array in both storage orders and compares them with MATLAB indexing of the same array.

This conflicts with #923, which restructures HDF5LazyArray/load_mat_style.m. That file's read code now lives in readSubscripts, which is where #923's grouped reads belong; the reshaping it moves into reshapeLoadedData is now in the base class.

🤖 Generated with Claude Code

Each lazy array turned MATLAB subscripts into reads itself. The Zarr v3
lazy array re-implemented HDF5LazyArray's checks, expected-size rules
and reordering, added its own choice between a bounding-box read and
element reads, and reversed dimensions in several places.

io.backend.base.LazyArray now implements load_mat_style: it checks the
subscripts, reads each dimension's indices once, sorted, and returns
the values in the order and shape the subscripts ask for, as a table
for a compound dataset. A backend implements two reads in its own
storage order: readSubscripts, which reads every combination of
per-dimension indices, and readPoints, which reads single elements. A
linear selection is read through its bounding box or element by
element, as getBoundingBoxLimits allows. A backend whose storage order
is the reverse of MatNWB's says so in isStorageOrderReversed, and the
base class reverses subscripts and dimensions for it. readHyperslab
serves start, count and stride through the same path.

HDF5LazyArray implements the two reads with its existing union-of-
hyperslabs and point selections; a linear selection that fills its
bounding box is now read as one hyperslab in any rank.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ehennestad
ehennestad added this pull request to stack #946 October 14, 2026 17:12
@ehennestad ehennestad mentioned this pull request Oct 14, 2026
2 of 3 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant