Skip to content

Key state resets on the message layout, via baseproc's axis-aware default - #233

Merged
cboulay merged 1 commit into
devfrom
cboulay/axis-aware-state-hash
Sep 4, 2026
Merged

Key state resets on the message layout, via baseproc's axis-aware default#233
cboulay merged 1 commit into
devfrom
cboulay/axis-aware-state-hash

Conversation

@cboulay

@cboulay cboulay commented Sep 4, 2026

Copy link
Copy Markdown
Member

First of three, each reviewable on its own:

  1. this one — adopt ezmsg-baseproc 1.12.0's axis-aware state hash
  2. Concat: cache the axes that describe the stream — two bugs the chunk_dim field newly lets us fix
  3. Downsample: the dimension is not configurable — a breaking settings change

What this fixes

ezmsg-baseproc 1.12.0's default _hash_message folds in the message key, the dims, the length of every dimension except the chunk dimension, the coordinate values on those dimensions, and the gain and offset of any linear axis among them.

23 of the 46 stateful processors here were hashing a strict subset of that. What they were all missing is the channel fingerprint: a source that renames or reorders channels without changing how many it sends — a device reconfigured mid-session, a montage swapped — looked identical, so per-channel state carried onto channels it did not belong to.

For a filter that state is numeric, and the result is not subtle:

first 4 samples of the new channel 'armB-1':
  filter state carried from armA: [-5.919  -4.772  -0.5114  1.064]
  with a correct reset:           [ 0.     -0.0013 -0.0023 -0.0026]
  max |difference| = 11.12   vs new-data amplitude 0.024

Nothing in the output announces it. tests/unit/test_state_reset_semantics.py reproduces that case end to end.

How the deletions were decided

Not by reading. A harness ran every override and the default over one stream and compared their reset points; an override was removed only where the default's set was a superset:

override                                    its resets      default resets   verdict
butterworthfilter.ButterworthFilter         [0,5,6,7,8]     [0,3,5,6,7,8]    DELETABLE (default also resets at [3])
slicer.SlicerTransformer                    [0,5,6,7,8]     [0,3,5,6,7,8]    DELETABLE (default also resets at [3])
spectrum.SpectrumTransformer                [0,3,5,6,7,8,9] [0,3,5,6,7,8]    KEEP -- default misses [9]
...
  3 = RELABEL at fixed channel count      9 = transform-length change

Four overrides stay, each for something the default cannot know:

processor why
align two alternating input streams; the default's key sensitivity resets it every message
spectrum the FFT is sized by the chunk dimension — the one length the default deliberately ignores — folded back via extra=
filterbank, wavelets dtype

Three get a narrower hash than the default, because their state genuinely depends on less. AdaptiveStandardScaler owns no arrays at all — its two child EWMATransformers hash themselves, and after a channel relabel its output is bit-identical (0.000e+00) to a scaler that only ever saw the new channels. Downsample and FilterbankDesign derive everything from one axis' gain.

chunk_dim

Declared wherever this package creates or consumes one: Window names its new axis, Spectrum and Aggregate clear it when they consume the dimension, Flatten follows a renamed preserve axis, Concat carries it. Without it the default has to guess, and a wrong guess is not a small error:

post-Window (win, time, ch)   0=first  1,2=win-count jitter  3=relabel  4=same  5=window-len change
  consumer assumes "time":  resets at [0, 1, 2, 3]
  message declares "win":   resets at [0, 3, 5]      correct

Fingerprint priming

Coordinate axes this package builds are handed over primed, via a new util.message.with_fingerprint. The digest is cached on the axis and pickles with it, so computing it at construction spares the first consumer in every receiving process from recomputing it on every message — unpickling builds a new axis object per message, so a cold axis is re-checksummed forever.

Deliberately not primed: coordinate axes along the chunk dimension. Their values are per-message and no consumer reads their fingerprint. There's a test pinning that so nobody "fixes" it.

Two tests now assert the opposite

test_common_rereference_field_values_change_is_not_detected and flatten's test_labels_outside_flatten_axes_do_not_reset, both deliberately. Each documented a concession made to avoid an O(bytes) per-message cost; CoordinateAxis.fingerprint removes that cost by computing the digest once per axis object rather than once per consumer, so the concession is no longer worth making.

Also

tests/helpers/recycled_shm.py had a latent gap of its own — _detach didn't copy axes, so it could not have detected a retained axis. Fixed, and verified it now catches one.

Performance

Measured three ways (benchmarks/benchmark_hash_overhead.py, --arm before|after|naive). Per _hash_message call, 256-channel ChannelMap: 0.22 µs → 0.45 µs warm. Across a 5-stateful-stage chain: 1.66 → 3.95 µs per message, 0.4% → 0.9% of a 415 µs chain.

The fingerprint is what keeps it there. Without it — the same correctness implemented the obvious way — it would be 8.29 µs and 6 checksums per message instead of 1.

End-to-end throughput is unchanged: 412 vs 420 µs/message against a 31–55 µs round-to-round spread, so the difference isn't resolvable by differencing chain runs. The benchmark says so rather than printing a number it can't back.

Testing

4247 passed, up from 4184.

Dependencies

ezmsg 3.10.0b2 and ezmsg-baseproc 1.12.0. The ezmsg pin is a pre-release until 3.10.0 ships.

🤖 Generated with Claude Code


On CI: this stack now sits on top of #236, which fixes test_mlx_cache_limit_actually_bounds_the_cache -- a test that had failed on every run since it landed on 2026-08-24, including main on 2026-08-25 with the byte-identical assertion, before any of this work. Nothing in this stack touches test_asarray.py or asarray.py. Merge order: #236, then this, then #234, then #235.

…ault

ezmsg-baseproc 1.12.0's default `_hash_message` folds in the message key, the
dims, the length of every dimension except the chunk dimension, the coordinate
*values* on those dimensions, and the gain and offset of any linear axis among
them. 23 of the 46 stateful processors here were hashing a strict subset of
that and now inherit it; 3 keep an override for something the default cannot
see; 1 keeps one for a reason the default gets wrong.

What the deletions buy is the channel fingerprint. A source that renames or
reorders channels without changing how many it sends -- a device reconfigured
mid-session, a montage swapped -- was invisible to these hashes, so per-channel
state carried onto channels it did not belong to. For a filter that state is
numeric, and the result is not subtle:

    first 4 samples of the new channel 'armB-1':
      filter state carried from armA: [-5.919  -4.772  -0.5114  1.064]
      with a correct reset:           [ 0.     -0.0013 -0.0023 -0.0026]
      max |difference| = 11.12   vs new-data amplitude 0.024

Nothing in the output announces it. `tests/unit/test_state_reset_semantics.py`
pins the behaviour that fell out, including that same filter case reproduced
end to end.

Each deletion was checked rather than assumed: a harness ran every override and
the default over one stream and compared their reset points, and an override
was only removed where the default's set was a superset. The four kept:

* `align` -- two alternating input streams; the default's key sensitivity would
  reset it on every message.
* `spectrum` -- the FFT is sized by the chunk dimension, the one length the
  default deliberately ignores, so it folds it back in via `extra=`.
* `filterbank`, `wavelets` -- dtype, which the default cannot see.

Three processors gain a *narrower* hash than the default, because their state
genuinely depends on less: `AdaptiveStandardScaler` owns no arrays (its two
child EWMATransformers hash themselves, verified bit-identical after a
relabel), and `Downsample` and `FilterbankDesign` derive everything from one
axis' gain.

`AxisArray.chunk_dim` is declared where this package creates or consumes one:
`Window` names its new axis, `Spectrum` and `Aggregate` clear it when they
consume the dimension, `Flatten` follows a renamed preserve axis, `Concat`
carries it. Without that the default has to guess, and a wrong guess is not a
small error -- it either thrashes on chunk-size jitter or stops noticing real
changes.

Coordinate axes this package builds are handed over primed, via a new
`util.message.with_fingerprint`. The digest is cached on the axis and pickles
with it, so computing it at construction spares the first consumer in every
receiving process from recomputing it on every message -- unpickling builds a
new axis object per message, so a cold axis is re-checksummed forever.
Deliberately not primed: coordinate axes along the chunk dimension, whose
values are per-message and whose fingerprint no consumer reads.

Two tests now assert the opposite of what they did, both deliberately:
`test_common_rereference_field_values_change_is_not_detected` and flatten's
`test_labels_outside_flatten_axes_do_not_reset`. Both documented a concession
made to avoid an O(bytes) per-message cost; `CoordinateAxis.fingerprint`
removes that cost by computing the digest once per axis object rather than once
per consumer, so the concession is no longer worth making.

`tests/helpers/recycled_shm.py` had a latent gap of its own: `_detach` did not
copy axes, so it could not have detected a retained axis. Fixed, and verified
that it now catches one.

Requires ezmsg 3.10.0b2 and ezmsg-baseproc 1.12.0.

4247 passed, up from 4184.
@cboulay
cboulay force-pushed the cboulay/axis-aware-state-hash branch from eaa86b7 to 3a56c4d Compare September 4, 2026 05:39
@cboulay
cboulay merged commit 8cf326e into dev Sep 4, 2026
14 checks passed
@cboulay
cboulay deleted the cboulay/axis-aware-state-hash branch September 4, 2026 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant