Key state resets on the message layout, via baseproc's axis-aware default - #233
Merged
Conversation
This was referenced Sep 4, 2026
cboulay
force-pushed
the
cboulay/axis-aware-state-hash
branch
from
September 4, 2026 04:52
070098e to
bc3ea44
Compare
cboulay
force-pushed
the
cboulay/axis-aware-state-hash
branch
from
September 4, 2026 05:00
bc3ea44 to
eaa86b7
Compare
This was referenced Sep 4, 2026
…ault
ezmsg-baseproc 1.12.0's default `_hash_message` folds in the message key, the
dims, the length of every dimension except the chunk dimension, the coordinate
*values* on those dimensions, and the gain and offset of any linear axis among
them. 23 of the 46 stateful processors here were hashing a strict subset of
that and now inherit it; 3 keep an override for something the default cannot
see; 1 keeps one for a reason the default gets wrong.
What the deletions buy is the channel fingerprint. A source that renames or
reorders channels without changing how many it sends -- a device reconfigured
mid-session, a montage swapped -- was invisible to these hashes, so per-channel
state carried onto channels it did not belong to. For a filter that state is
numeric, and the result is not subtle:
first 4 samples of the new channel 'armB-1':
filter state carried from armA: [-5.919 -4.772 -0.5114 1.064]
with a correct reset: [ 0. -0.0013 -0.0023 -0.0026]
max |difference| = 11.12 vs new-data amplitude 0.024
Nothing in the output announces it. `tests/unit/test_state_reset_semantics.py`
pins the behaviour that fell out, including that same filter case reproduced
end to end.
Each deletion was checked rather than assumed: a harness ran every override and
the default over one stream and compared their reset points, and an override
was only removed where the default's set was a superset. The four kept:
* `align` -- two alternating input streams; the default's key sensitivity would
reset it on every message.
* `spectrum` -- the FFT is sized by the chunk dimension, the one length the
default deliberately ignores, so it folds it back in via `extra=`.
* `filterbank`, `wavelets` -- dtype, which the default cannot see.
Three processors gain a *narrower* hash than the default, because their state
genuinely depends on less: `AdaptiveStandardScaler` owns no arrays (its two
child EWMATransformers hash themselves, verified bit-identical after a
relabel), and `Downsample` and `FilterbankDesign` derive everything from one
axis' gain.
`AxisArray.chunk_dim` is declared where this package creates or consumes one:
`Window` names its new axis, `Spectrum` and `Aggregate` clear it when they
consume the dimension, `Flatten` follows a renamed preserve axis, `Concat`
carries it. Without that the default has to guess, and a wrong guess is not a
small error -- it either thrashes on chunk-size jitter or stops noticing real
changes.
Coordinate axes this package builds are handed over primed, via a new
`util.message.with_fingerprint`. The digest is cached on the axis and pickles
with it, so computing it at construction spares the first consumer in every
receiving process from recomputing it on every message -- unpickling builds a
new axis object per message, so a cold axis is re-checksummed forever.
Deliberately not primed: coordinate axes along the chunk dimension, whose
values are per-message and whose fingerprint no consumer reads.
Two tests now assert the opposite of what they did, both deliberately:
`test_common_rereference_field_values_change_is_not_detected` and flatten's
`test_labels_outside_flatten_axes_do_not_reset`. Both documented a concession
made to avoid an O(bytes) per-message cost; `CoordinateAxis.fingerprint`
removes that cost by computing the digest once per axis object rather than once
per consumer, so the concession is no longer worth making.
`tests/helpers/recycled_shm.py` had a latent gap of its own: `_detach` did not
copy axes, so it could not have detected a retained axis. Fixed, and verified
that it now catches one.
Requires ezmsg 3.10.0b2 and ezmsg-baseproc 1.12.0.
4247 passed, up from 4184.
cboulay
force-pushed
the
cboulay/axis-aware-state-hash
branch
from
September 4, 2026 05:39
eaa86b7 to
3a56c4d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First of three, each reviewable on its own:
chunk_dimfield newly lets us fixWhat this fixes
ezmsg-baseproc 1.12.0's default
_hash_messagefolds in the message key, the dims, the length of every dimension except the chunk dimension, the coordinate values on those dimensions, and the gain and offset of any linear axis among them.23 of the 46 stateful processors here were hashing a strict subset of that. What they were all missing is the channel fingerprint: a source that renames or reorders channels without changing how many it sends — a device reconfigured mid-session, a montage swapped — looked identical, so per-channel state carried onto channels it did not belong to.
For a filter that state is numeric, and the result is not subtle:
Nothing in the output announces it.
tests/unit/test_state_reset_semantics.pyreproduces that case end to end.How the deletions were decided
Not by reading. A harness ran every override and the default over one stream and compared their reset points; an override was removed only where the default's set was a superset:
Four overrides stay, each for something the default cannot know:
alignspectrumextra=filterbank,waveletsThree get a narrower hash than the default, because their state genuinely depends on less.
AdaptiveStandardScalerowns no arrays at all — its two childEWMATransformers hash themselves, and after a channel relabel its output is bit-identical (0.000e+00) to a scaler that only ever saw the new channels.DownsampleandFilterbankDesignderive everything from one axis' gain.chunk_dimDeclared wherever this package creates or consumes one:
Windownames its new axis,SpectrumandAggregateclear it when they consume the dimension,Flattenfollows a renamed preserve axis,Concatcarries it. Without it the default has to guess, and a wrong guess is not a small error:Fingerprint priming
Coordinate axes this package builds are handed over primed, via a new
util.message.with_fingerprint. The digest is cached on the axis and pickles with it, so computing it at construction spares the first consumer in every receiving process from recomputing it on every message — unpickling builds a new axis object per message, so a cold axis is re-checksummed forever.Deliberately not primed: coordinate axes along the chunk dimension. Their values are per-message and no consumer reads their fingerprint. There's a test pinning that so nobody "fixes" it.
Two tests now assert the opposite
test_common_rereference_field_values_change_is_not_detectedand flatten'stest_labels_outside_flatten_axes_do_not_reset, both deliberately. Each documented a concession made to avoid an O(bytes) per-message cost;CoordinateAxis.fingerprintremoves that cost by computing the digest once per axis object rather than once per consumer, so the concession is no longer worth making.Also
tests/helpers/recycled_shm.pyhad a latent gap of its own —_detachdidn't copy axes, so it could not have detected a retained axis. Fixed, and verified it now catches one.Performance
Measured three ways (
benchmarks/benchmark_hash_overhead.py,--arm before|after|naive). Per_hash_messagecall, 256-channel ChannelMap: 0.22 µs → 0.45 µs warm. Across a 5-stateful-stage chain: 1.66 → 3.95 µs per message, 0.4% → 0.9% of a 415 µs chain.The fingerprint is what keeps it there. Without it — the same correctness implemented the obvious way — it would be 8.29 µs and 6 checksums per message instead of 1.
End-to-end throughput is unchanged: 412 vs 420 µs/message against a 31–55 µs round-to-round spread, so the difference isn't resolvable by differencing chain runs. The benchmark says so rather than printing a number it can't back.
Testing
4247 passed, up from 4184.
Dependencies
ezmsg 3.10.0b2andezmsg-baseproc 1.12.0. The ezmsg pin is a pre-release until 3.10.0 ships.🤖 Generated with Claude Code
On CI: this stack now sits on top of #236, which fixes
test_mlx_cache_limit_actually_bounds_the_cache-- a test that had failed on every run since it landed on 2026-08-24, includingmainon 2026-08-25 with the byte-identical assertion, before any of this work. Nothing in this stack touchestest_asarray.pyorasarray.py. Merge order: #236, then this, then #234, then #235.