000044: incorrect rate data in SpatialSeries

[stephprince]

In this dandiset, the data in the behavior processing module indicates that the position tracking has a sampling rate of 0.0256 Hz. However, other information in the file suggests that this sampling rate should be the inverse, 39.0625 Hz.

See this example where the SpatialSeries has a starting time which matches the epochs table MazeEpoch start time, but has an end time ~36 days later instead of the end time ~35 minutes later that is suggested by the epochs table MazeEpoch stop time.

Other Buzsaki lab files (e.g. dandiset 000067) generated around the same time also have position tracking data acquired at a sampling rate of 39.0625 Hz.

[bendichter]

I went through all eight assets in 000044 and can confirm this. In every session the rate attribute on both SpatialSeries objects is approximately 0.0256 and its reciprocal is 39.0625 Hz, so the stored value is the sampling period rather than the frequency.

session	position samples	stored rate	1/rate (Hz)
Achilles-10252013	80,762	0.025599911	39.0626
Achilles-11012013	104,457	0.025600008	39.0625
Buddy-06272013	90,942	0.025599833	39.0628
Cicero-09012014	217,235	0.025599926	39.0626
Cicero-09102014	118,360	0.025599865	39.0627
Cicero-09172014	120,227	0.025599907	39.0626
Gatsby-08022013	91,215	0.025599956	39.0626
Gatsby-08282013	105,180	0.025599924	39.0626

One detail that may help whoever applies the fix: the agreement with the epochs table is exact rather than approximate. In all eight files starting_time equals the MazeEpoch start time, and starting_time + n_samples * rate equals the MazeEpoch stop time to within 2e-7 seconds. The stored value looks like it was computed as the maze duration divided by the number of position samples, which is why it differs slightly from 1/39.0625 in each session. So the corrected rate is just the reciprocal of the stored value, and it should not be replaced with a hard-coded 39.0625 if the intent is to preserve the existing sample timing.

I will also mention one piece of evidence that looks convincing but is not. The unit attribute on starting_time reads "seconds", which at first glance seems to confirm that the value is a period. It does not, because that attribute is a fixed value in the NWB schema describing the units of starting_time itself, and pynwb writes it regardless of what rate contains. The reciprocal landing on 39.0625 Hz and the agreement with the epochs table are the actual evidence.

Separately, while checking this I ran into an unrelated problem in the same dandiset, in the units table rather than the behavior module. In three of the eight sessions (Buddy-06272013, Cicero-09012014, Gatsby-08022013) the first element of /units/spike_times_index is short by an exact multiple of 65,536, which looks like a 16-bit wraparound. The effect is that the first unit's spike train is truncated and its missing tail is prepended to the row for the second unit, whose spike times therefore jump backwards part way through. The spike times themselves are intact and in order, so correcting the single index value repairs both rows, and I verified that every row in the file is monotonic afterwards. This is worth flagging because the truncated unit passes a monotonicity check and in two of the three sessions ends up with no spikes at all inside the MazeEpoch, so it would look like a well-formed cell that is simply silent during behavior. I mention it here only because it is the same dandiset, and I am happy to open a separate discussion with the full details rather than mix it into this thread.

[bendichter]

All eight assets have been repaired and re-uploaded to the 000044 draft. Scripts and the full write-up are here: https://gist.github.com/bendichter/97ada215c394a508b2bf7d11872bfff1

The rate fix

Confirmed as diagnosed: rate held the sampling period, and the corrected value is the exact reciprocal of the stored one rather than a hard-coded 39.0625. Each session's stored value looks like maze_duration / n_samples, so the reciprocal preserves the original sample timing, and the file's own epochs table is the authority on what that timing was. The per-session rates below are all slightly different, which is the signature of having preserved that timing rather than overwritten it with a nominal value.

Verified by streaming each asset back from the archive after upload, not from local disk:

session	rate (Hz)	end err vs MazeEpoch stop	spike_times_index[0]
Achilles-10252013	39.062636	-5.01e-08 s	34,006	PASS
Achilles-11012013	39.062488	+1.48e-08 s	16,133	PASS
Buddy-06272013	39.062755	+6.23e-08 s	80,214	PASS — wrap +65,536
Cicero-09012014	39.062612	-1.55e-07 s	73,852	PASS — wrap +65,536
Cicero-09102014	39.062706	+1.94e-07 s	26,454	PASS
Cicero-09172014	39.062642	+2.86e-08 s	19,758	PASS
Gatsby-08022013	39.062567	+9.94e-08 s	193,036	PASS — wrap +131,072
Gatsby-08282013	39.062616	+1.80e-07 s	28,091	PASS

The spike_times_index wraparound

One correction to my earlier comment, which said the first index element was short by a multiple of 65,536 in three sessions. That is right, but the multiple is not always 1 — Gatsby-08022013 was short by 2 × 65,536 = 131,072. The script therefore derives the true boundary by locating the backward jump in unit 1's row, and refuses to write unless the resulting deficit is an exact multiple of 65,536 and falls strictly inside that row. Anyone applying a blanket +65536 would silently under-repair that session.

After the fix, every unit row in every file is monotonic, and spike_times_index[-1] == len(spike_times) throughout. In Buddy, unit 0 goes from 14,678 to 80,214 spikes and from 0 to 2,947 spikes inside the MazeEpoch — so it stops looking like a well-formed cell that happens to be silent during behavior.

Subject metadata (not part of this issue, but it blocked the upload)

The species string Rattus norvegicus domestica - Long Evans cannot be resolved by dandi-cli's species_map (its Rattus entry has no prefix rule), and that failure happens during metadata extraction — before any --validation flag applies, so it hard-blocks upload regardless. Normalized to Rattus norvegicus, with sex → M, weight → 0.3 kg, and age → /. Every original value is preserved verbatim in Subject.description (Rattus norvegicus domestica - Long Evans; weight range 250-350g), including the midpoint weight, which is an interpolation the source data does not state.

The strain would normally belong in Subject.strain, but these files cache NWB schema 2.2.5, which has no strain field — writing one would violate the file's own cached schema.

Two things left alone deliberately

electrodes/filtering fails validation with incorrect type - expected 'float32', got 'utf'. This is an upstream defect in schema 2.2.5, which declares dtype: float32 for a field its own docstring calls "Description of hardware filtering." The files store the semantically correct thing — the text 'none'. The column is required in 2.2.5 (no quantity key), so deleting it is not an option either; the only way to satisfy the declaration would be to destroy correct information. Upstream removed these hard-coded electrodes columns entirely, so the conflict disappears on migration. units.resolution is unset, which is a best-practice note rather than an error.

Publishing is blocked, and not by these files

The corrected data is in draft. Cutting a published version currently fails:

POST /api/dandisets/000044/versions/draft/publish/  ->  HTTP 400
asset_validation_errors: 0
version_validation_errors: 1
  doi: '10.48324/dandi.123456/0.123456.1234' does not match
       '^10.80507/dandi\.\d{6}/\d+\.\d+\.\d+'

asset_validation_errors: 0 — the NWB files are clean. The server enforces DANDI's new 10.80507 DOI prefix, while the DOI minted for this dandiset uses the old 10.48324 prefix. The quoted string is a placeholder generated to test DOI shape; the draft metadata contains no doi field to correct, so it is not fixable client-side. 000067 fails identically, while dandisets last published under the new prefix validate fine — so this looks like it affects any dandiset last published under the old prefix.

Until a new version can be published, 0.250624.0426 still serves the uncorrected rate. Note also that re-upload mints new asset IDs, so the neurosift links in the opening post continue to resolve to the old published version rather than the fixed draft.