Let the base class decide when these transformers reset - #27
Merged
Conversation
ezmsg-baseproc 1.12.0's default `_hash_message` folds in the message key, the
dims, the length of every dimension except the chunk dimension, the coordinate
*values* on those dimensions, and the gain and offset of any linear axis among
them. Four of the five overrides here were subsets of that and are gone.
Checked empirically rather than by reading, against a `(win, time, ch)` stream
where each step changes exactly one property:
adaptive_decomp own [0,1,2,6,7] default [0,3,5,6,7]
flatten own [0,1,2,5,6] default [0,3,5,6,7]
ssr own [0,6,7] default [0,3,5,6,7]
mlp_old key + n_ch only, a strict subset
(0 first 1,2 win-count jitter 3 relabel 5 window length 6 channel count
7 new key)
Two of them were resetting on *win-count jitter* -- the chunk dimension's
extent, which is just how much arrived -- while missing a channel relabel.
`adaptive_decomp` had it exactly backwards: `settings.axis` defaults to
`"!time"`, so on a windowed message its `iter_axis` resolved to `time` and it
excluded the window length while including the window count. Every one of the
four now notices a relabel, which none of them did.
`ssr` also drops `group_spec_fingerprint`, whose O(1) "bank field present"
boolean was a deliberate concession to avoid a per-message cost that scaled
with channel count. `CoordinateAxis.fingerprint` removes that cost by
computing the digest once per axis object rather than once per consumer, so
the concession is no longer worth making. Its test asserted the old behaviour
by name and now asserts the new.
Two overrides stay, for reasons the default cannot cover:
* `adaptive_linear_regressor` has nothing to reset, so a constant is correct
and cheapest. Changed from -1 to 0: -1 is the sentinel `_hash` starts at and
that `_request_reset()` writes, so returning it made an explicitly requested
reset compare equal and be swallowed.
* `sgd` needs a constant for a different reason, and this one was a live
regression. Its `partial_fit` sets `_hash = 0` itself; training samples
arrive as `(time, ch, freq)` and inference windows as `(win, time, ch, freq)`,
so any layout-derived hash differs between the two and every alternation ran
`_reset_state`, which calls `_refreshed_model()` and throws the fitted model
away. `test_sgd` caught it. The assumption was previously inherited from
baseproc's old constant default and is now stated where it is relied on.
`FlattenTransformer` gains `STREAMING_DIMS = ("win",)`. Its canonical input is
`(win, time, ch)`, where `win` grows per message and `time` is the lag
dimension whose length sizes the lag axis; the base class's `("time",)`
fallback excludes exactly the wrong one. This is load-bearing rather than
cosmetic, because released ezmsg-sigproc does not yet set `chunk_dim`, so
these messages arrive undeclared.
Coordinate axes built here are now primed via a new `util.with_fingerprint`:
the digest is cached on the axis and pickles with it, so computing it at
construction spares the first consumer in every receiving process from
recomputing it on every message. Ten sites across eight modules. The dummy
message in `sklearn.fit` is left alone -- it drives `_reset_state` and is never
emitted.
Requires ezmsg-baseproc 1.12.0. The direct `ezmsg` pin is raised to 3.10.0b2
alongside it: uv only enables pre-releases for a package named with a
pre-release marker in *this* file, so leaving it to baseproc's transitive
requirement fails resolution outright.
369 passed, up from 360. The 9 new tests are mutation-checked: dropping the
priming fails 6, and reverting STREAMING_DIMS to ("time",) fails 3.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ezmsg-baseproc 1.12.0's default
_hash_messagefolds in the message key, the dims, the length of every dimension except the chunk dimension, the coordinate values on those dimensions, and the gain and offset of any linear axis among them. Four of the five overrides here were subsets of that.Checked empirically, not by reading
A harness ran each override and the default over a
(win, time, ch)stream where each step changes exactly one property — an earlier version reset the window count while relabelling and made the relabel column meaningless:Two were resetting on win-count jitter — the chunk dimension's extent, which is just how much arrived — while missing a channel relabel.
adaptive_decomphad it exactly backwards:settings.axisdefaults to"!time", so on a windowed message itsiter_axisresolved totimeand it excluded the window length while including the window count.All four now notice a relabel, which none of them did.
ssralso dropsgroup_spec_fingerprint, whose O(1) "bank field present" boolean was a deliberate concession to avoid a per-message cost that scaled with channel count.CoordinateAxis.fingerprintremoves that cost by computing the digest once per axis object rather than once per consumer, so the concession is no longer worth making. Its test asserted the old behaviour by name and now asserts the new.Two overrides stay
adaptive_linear_regressorhas nothing to reset, so a constant is correct and cheapest. Changed from-1to0:-1is the sentinel_hashstarts at and that_request_reset()writes, so returning it made an explicitly requested reset compare equal and be silently swallowed.sgdneeds a constant for a different reason, and this one was a live regression caught by the existingtest_sgd. Itspartial_fitsets_hash = 0itself; training samples arrive as(time, ch, freq)and inference windows as(win, time, ch, freq), so any layout-derived hash differs between the two and every alternation ran_reset_state— which calls_refreshed_model()and throws the fitted model away. The assumption was previously inherited from baseproc's old constant default and is now stated where it is relied on.FlattenTransformer.STREAMING_DIMS = ("win",)Its canonical input is
(win, time, ch), wherewingrows per message andtimeis the lag dimension whose length sizes the lag axis. The base class's("time",)fallback excludes exactly the wrong one.This is load-bearing, not cosmetic: released ezmsg-sigproc does not yet set
chunk_dim, so these messages arrive undeclared and the fallback is what decides.Fingerprint priming
Coordinate axes built here are primed via a new
util.with_fingerprint— ten sites across eight modules. The digest is cached on the axis and pickles with it, so computing it at construction spares the first consumer in every receiving process from recomputing it on every message; unpickling builds a new axis object per message, so a cold axis is re-checksummed forever.The dummy message in
sklearn.fitis left alone — it drives_reset_stateand is never emitted.Testing
369 passed, up from 360. The 9 new tests are mutation-checked: dropping the priming fails 6, and reverting
STREAMING_DIMSto("time",)fails 3.Known limitation
FlattenTransformerraises on a message that declareschunk_dim, because it renameswin→timeand the inner ezmsg-sigproc transformer doesn't carry the field across. That's fixed in sigproc, not yet released, so it resolves when that ships. The tests here use undeclared messages, which is what this package actually receives today.Dependencies
ezmsg-baseproc>=1.12.0, and the directezmsgpin raised to3.10.0b2alongside it — uv only enables pre-releases for a package named with a pre-release marker in this file, so leaving it to the transitive requirement fails resolution outright.🤖 Generated with Claude Code