fix(dynamictable): read ragged columns once per index level in getRow

[ehennestad]

fix #787

Motivation

Makes toTable and getRow fast on ragged columns read from a file, which were read one row at a time: a 20-unit Units table with waveforms takes 0.25 s instead of 29.6 s.

Background — A ragged column stores rows of different lengths as one flat data array plus an index of cumulative row ends. In a Units table, waveforms is doubly ragged: one index has a row per unit that points at the unit's spikes, and a second index has a row per spike that points at its waveform. Units with 3 and 2 spikes give the unit index [3 5].

Problem — A user reads an NWB file and calls toTable or getRow on a Units table with waveforms. The call takes minutes, while the same call on the table before export takes about a second. In the 20-unit example below (26,583 spikes), toTable takes 0.94 s before export and 29.6 s after reading the file. For the full table in #787 (172 units, 211,722 spikes) it takes 181 s here, and the reporter measured 1792 s. The cause is that each unit's index entries and each spike's waveform are read from the file one at a time, so 26,583 spikes cost more than 26,000 separate file reads. The output is correct, so the only symptom is the wait.

Solution — getRow now reads only the elements of the requested rows, and for contiguous rows, as in toTable, it reads each level of the column once. The rows are then taken from those reads in memory. The example takes 0.25 s after reading the file, and the output is unchanged.

Additional fix: rows of a column whose dataset has no elements

Problem — A user reads a file in which every row of a ragged column is empty, so the column's dataset holds no elements, and getRow or toTable fails. In the second example below, getRow(1:2) on a table whose amplitudes dataset was written as an extendable dataset with nothing appended stops with DataStub indices for dimension 1 must be less than or equal to dimension size 0 instead of returning two empty rows. The cause is that an empty row is read as an empty selection from the column, and the reader reads the first element of the dataset to learn the type of an empty selection; a dataset with no elements has no first element. This also fails on main.

Solution — When the dataset has no elements, the empty row is built from the column's data type, so getRow(1:2) returns two 0x0 rows.

Why in this PR — This PR reads the empty rows of a column through one new helper, and the fix changes only that helper.

What changed

• getRow and toTable read only the elements of the requested rows of a ragged column. toTable and any contiguous range of rows cost one read per level, so a doubly ragged column such as waveforms costs 3 reads.
• For rows with gaps between them, such as getRow([1 172]), each index level and any one-dimensional data are still read in one call. Multi-dimensional data such as waveforms is read in one call per contiguous run of requested elements, so a scattered selection never makes more reads than before this PR.
• The output is unchanged for columns in memory, read from file, and stored in a DataPipe, including empty rows.
• A ragged column whose dataset has no elements returns empty rows instead of an error.

Implementation notes

• select in +types/+util/+dynamictable/getRow.m looks up the column's VectorData and VectorIndex objects once. For a ragged column, getRaggedRows reads each index level in one call for exactly the index elements the requested rows need (readElementRanges), expands those rows' element ranges into the rows requested from the level below (expandRanges), and reads the data with readDataRows. It then nests the rows level by level with mat2cell.
• readDataRows reads one-dimensional data in one call, as a point selection. It reads multi-dimensional data in one call per contiguous run, because a multi-dimensional selection with gaps goes through io.space.findShapes, whose cost grows faster than linearly with the number of irregular runs: 1,000 runs took 5.7 s on the machine used for this PR.
• A scattered selection of many short rows in a multi-dimensional column therefore costs one read per run. Every other row of a 5,000-row 8 x n column takes 3.2 s, the same as main. Reading the span that covers the rows takes 0.07 s but also loads the rows that were not requested. A setting to choose between time and memory could be added later.
• An empty row at the data level comes from one empty-selection read of the column, not from a block of rows already read. This keeps the type and shape a direct read gives it: today an empty row reads as 0x0 from file and 0x1 in memory, and this PR leaves that as it is.
• When a file-backed column's dataset has no elements, readEmptyRow builds the empty row from the column's dataType instead of reading. The types follow io.parseCompound, which builds the columns of a compound dataset without rows the same way: text and non-boolean enums as cell arrays, booleans as logical arrays, numeric and reference types from <class>.empty, and a compound type as a table with one variable per member.
• getRaggedRows takes the chain of column objects rather than the table, so it can move onto VectorIndexBase.getRows from Refactor plan: consolidate DynamicTable helpers around column behaviour, one column accessor, declared structure and pure validation #907 without changes.
• readRows, orientRows, getRowDimension and indexRows are the previous data branch split into parts, so reading from the column and slicing from a window use the same rule for which axis holds the rows.
• The read tests use a spy, tests.unit.io.backend.doubles.HDF5LazyArraySpy, that subclasses HDF5LazyArray and is passed to the DataStub constructor, because DataStub is sealed. It counts the reads and records the elements each read selects.

Examples

toTable on a Units table with waveforms, read from file

The snippet builds a Units table with 20 units and one 64-sample waveform per spike, exports it, reads it back, and times toTable on both copies. It is the script from #787 with fewer units.

rng(787)
numUnits = 20;
spikesPerUnit = randi([500, 2000], 1, numUnits);
numSpikes = sum(spikesPerUnit);

waveforms = types.hdmf_common.VectorData( ...
    'description', 'one 64-sample waveform per spike', ...
    'data', rand(64, numSpikes));
waveformsIndex = types.hdmf_common.VectorIndex( ...
    'description', 'index into waveforms, one row per spike', ...
    'data', uint64(1:numSpikes)', ...
    'target', types.untyped.ObjectView(waveforms));
waveformsIndexIndex = types.hdmf_common.VectorIndex( ...
    'description', 'index into waveforms_index, one row per unit', ...
    'data', uint64(cumsum(spikesPerUnit))', ...
    'target', types.untyped.ObjectView(waveformsIndex));

nwb = NwbFile('session_description', 'issue 787', 'identifier', 'issue787', ...
    'session_start_time', datetime(2018, 4, 25, 'TimeZone', 'local'));
nwb.units = types.core.Units('description', 'units', 'colnames', {'waveforms'}, ...
    'waveforms', waveforms, 'waveforms_index', waveformsIndex, ...
    'waveforms_index_index', waveformsIndexIndex);
nwbExport(nwb, 'issue787.nwb');
nwbIn = nwbRead('issue787.nwb', 'ignorecache');

tic; memoryTable = nwb.units.toTable(); memorySeconds = toc;
tic; fileTable = nwbIn.units.toTable(); fileSeconds = toc;
fprintf('spikes: %d\n', numSpikes);
fprintf('toTable in memory: %.2f s\n', memorySeconds);
fprintf('toTable after read: %.2f s\n', fileSeconds);
fprintf('tables equal: %d\n', isequal(memoryTable, fileTable));

Before — Reading the table from the file takes about 30 times as long as building it from memory. tables equal: 1 shows that the result was already correct; only the time is wrong.

spikes: 26583
toTable in memory: 0.94 s
toTable after read: 29.55 s
tables equal: 1

After — Reading from the file takes no longer than building the table in memory, and the two tables are still equal.

spikes: 26583
toTable in memory: 0.21 s
toTable after read: 0.25 s
tables equal: 1

Timings are from one machine and vary with platform and disk.

A ragged column whose dataset has no elements

The snippet writes a table with two rows and a ragged amplitudes column. The column's data is an extendable dataset with nothing appended, so both rows are empty and the dataset holds no elements. It then reads the file and gets both rows.

amplitudes = types.hdmf_common.VectorData('description', 'amplitudes', ...
    'data', types.untyped.DataPipe('maxSize', Inf, 'dataType', 'double'));
amplitudesIndex = types.hdmf_common.VectorIndex('description', 'index into amplitudes', ...
    'data', uint64([0; 0]), 'target', types.untyped.ObjectView(amplitudes));
nwb = NwbFile('session_description', 'empty rows', 'identifier', 'empty_rows', ...
    'session_start_time', datetime(2024, 1, 1, 'TimeZone', 'local'));
nwb.acquisition.set('rows_without_amplitudes', types.hdmf_common.DynamicTable( ...
    'description', 'table whose rows have no amplitudes', 'colnames', {'amplitudes'}, ...
    'amplitudes', amplitudes, 'amplitudes_index', amplitudesIndex, ...
    'id', types.hdmf_common.ElementIdentifiers('data', int64([0; 1]))));
nwbExport(nwb, 'empty_rows.nwb');
nwbIn = nwbRead('empty_rows.nwb', 'ignorecache');
tableIn = nwbIn.acquisition.get('rows_without_amplitudes');

rows = tableIn.getRow(1:2);
fprintf('rows: %d\n', height(rows));
fprintf('row 1 amplitudes: %s, size %s\n', class(rows.amplitudes{1}), strjoin(string(size(rows.amplitudes{1})), 'x'));

Before — getRow fails while reading the empty rows, so no rows are returned.

Error: DataStub indices for dimension 1 must be less than or equal to dimension size 0

After — Both rows are returned, each with an empty double array, which is how an empty row of a file-backed numeric column reads.

rows: 2
row 1 amplitudes: double, size 0x0

How to test

Run the snippet above on main and on this branch and compare the toTable after read line. Then run the new tests. testGetRowReadsEachLevelOnce fails on main and passes here. testGetRowReadsOnlyRequestedElements checks that getRow([1 4]) does not read the waveforms of unit 3, which lies between the two. testGetRowWhenColumnHasNoElements covers the additional fix; it fails on main with the error above. The other tests check that file-backed, in-memory and DataPipe columns return the same rows as before.

results = runtests('tests.unit.DynamicTableRaggedReadTest');
disp(table(results))

Checklist

• Have you ensured the PR description clearly describes the problem and solutions?
• Have you checked to ensure that there aren't other open or previously closed Pull Requests for the same change?
• If this PR fixes an issue, is the first line of the PR description fix #XX where XX is the issue number?

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 97.22222% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.52%. Comparing base (0bb63d3) to head (2fef956).

Files with missing lines	Patch %	Lines
+types/+util/+dynamictable/getRow.m	97.22%	4 Missing ⚠️

Additional details and impacted files

@@                     Coverage Diff                     @@
##           rename-getrow-variables     #909      +/-   ##
===========================================================
+ Coverage                    95.44%   95.52%   +0.07%     
===========================================================
  Files                          240      240              
  Lines                         8870     8955      +85     
===========================================================
+ Hits                          8466     8554      +88     
+ Misses                         404      401       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.