Dejitter clock reconstruction for jittery NWB timestamps

[cboulay]

Problem

Some acquisition streams carry per-sample timestamps that are correct in aggregate (their median interval gives the true sample rate) but jittery sample-to-sample — the pipeline stamps each sample at arrival time rather than from the ADC clock. The iterator's gap-splitter then shatters such a stream into tens of thousands of tiny messages. On the reference file, an 8.2M-sample HUB1 stream fragments into ~285k messages (median run length 13 samples), which makes offline processing intractable.

The jitter is not data loss — the same structure appears in the device timestamps, and the two acquisition hubs share a PTP-synchronised device clock — so the fix is to replace the jittery timestamps with a smoothed, strictly-monotone version that preserves the true (slowly drifting, non-linear) clock while discarding the jitter, keeping the converted clock so cross-source alignment is retained.

Approach

New clockmodel.py (pure numpy, file-agnostic — usable both at read time and by a future offline NWB-corrector):

• Monotone piecewise-linear map through binned-median knots, numpy-only (no scipy), pluggable method= for a future PCHIP. End segments are extrapolated to the true domain edges so the outer half-bins aren't clamped flat.
• Self-fit reconstruction (index → dataset_time) for a lone stream.
• Shared-clock model (the PTP idea): build the device → dataset conversion C from the cleanest group member, clean the target's device time from its sample index, push through C, and re-anchor to the target's own fixed latency via a robust median. A jittery stream is rescued by a clean sibling.
• Auto clock-group detection from device-time endpoints (cheap scalar reads) + disk cache keyed by file signature and fit params for multi-pass workloads.
• Real-gap guard: genuine data gaps are detected (threshold auto-derived above each stream's jitter envelope, verified as sustained level shifts) and preserved per-segment, so the iterator still splits a chunk there while jitter is smoothed.

Integrated into NWBSlicer: pairs converted ↔ *_device_ts streams by HDF5 object identity (hard-link), transparently loads and then hides partners under a stream_keys filter, and recomputes the stop bound in float64 while excluding the epoch-scale device partners. Forwarded through the iterator and clock-driven producer via new settings — dejitter (default on), clock_groups, dejitter_cache, real_gap_threshold.

Results on the reference file

• HUB1: 284,159 gaps → 0, strictly monotone, within 0.58 ms std of the recorded timestamps (cross-hub shared model); HUB2: 27 → 0.
• Iterator fragments collapse from ~900/chunk to 1/chunk, identical chunk grid, zero sample loss.
• Cold open 1.13 s → warm (cached) 0.24 s.
• Real-gap guard auto-threshold lands at 13.4 ms on HUB1 (above its 7.9 ms jitter tail) → 0 false gaps, while a genuine larger gap is preserved.

Notable subtleties handled

• Endpoint clamping of binned-median knots (was mis-timing the outer half-bins by seconds).
• float32/epoch bound blow-up: *_device_ts epoch timestamps + a float32 rate under NumPy weak promotion inflated stop_time to ~1.7e9; and a knife-edge ceil at epoch scale could drop a boundary sample — both fixed.
• Real gaps vs clock-sync steps are indistinguishable from timestamps alone, so the guard is conservative by design (preserves only jumps well beyond the jitter tail); real_gap_threshold allows explicit control.

Tests

28 new tests (test_clockmodel.py unit + test_dejitter.py slicer/iterator integration, incl. a gapped NWB fixture verifying gap preservation and the disable path). Full suite: 198 passed, ruff clean.

🤖 Generated with Claude Code