Skip to content

Let the base class decide when these transformers reset - #27

Merged
cboulay merged 1 commit into
devfrom
cboulay/baseproc-hashing
Sep 4, 2026
Merged

Let the base class decide when these transformers reset#27
cboulay merged 1 commit into
devfrom
cboulay/baseproc-hashing

Conversation

@cboulay

@cboulay cboulay commented Sep 4, 2026

Copy link
Copy Markdown
Member

ezmsg-baseproc 1.12.0's default _hash_message folds in the message key, the dims, the length of every dimension except the chunk dimension, the coordinate values on those dimensions, and the gain and offset of any linear axis among them. Four of the five overrides here were subsets of that.

Checked empirically, not by reading

A harness ran each override and the default over a (win, time, ch) stream where each step changes exactly one property — an earlier version reset the window count while relabelling and made the relabel column meaningless:

override            its resets       default resets
adaptive_decomp     [0,1,2,6,7]      [0,3,5,6,7]
flatten             [0,1,2,5,6]      [0,3,5,6,7]
ssr                 [0,6,7]          [0,3,5,6,7]
mlp_old             key + n_ch only, a strict subset

  0 first   1,2 win-count jitter   3 relabel   5 window length
  6 channel count   7 new key

Two were resetting on win-count jitter — the chunk dimension's extent, which is just how much arrived — while missing a channel relabel. adaptive_decomp had it exactly backwards: settings.axis defaults to "!time", so on a windowed message its iter_axis resolved to time and it excluded the window length while including the window count.

All four now notice a relabel, which none of them did.

ssr also drops group_spec_fingerprint, whose O(1) "bank field present" boolean was a deliberate concession to avoid a per-message cost that scaled with channel count. CoordinateAxis.fingerprint removes that cost by computing the digest once per axis object rather than once per consumer, so the concession is no longer worth making. Its test asserted the old behaviour by name and now asserts the new.

Two overrides stay

adaptive_linear_regressor has nothing to reset, so a constant is correct and cheapest. Changed from -1 to 0: -1 is the sentinel _hash starts at and that _request_reset() writes, so returning it made an explicitly requested reset compare equal and be silently swallowed.

sgd needs a constant for a different reason, and this one was a live regression caught by the existing test_sgd. Its partial_fit sets _hash = 0 itself; training samples arrive as (time, ch, freq) and inference windows as (win, time, ch, freq), so any layout-derived hash differs between the two and every alternation ran _reset_state — which calls _refreshed_model() and throws the fitted model away. The assumption was previously inherited from baseproc's old constant default and is now stated where it is relied on.

FlattenTransformer.STREAMING_DIMS = ("win",)

Its canonical input is (win, time, ch), where win grows per message and time is the lag dimension whose length sizes the lag axis. The base class's ("time",) fallback excludes exactly the wrong one.

This is load-bearing, not cosmetic: released ezmsg-sigproc does not yet set chunk_dim, so these messages arrive undeclared and the fallback is what decides.

Fingerprint priming

Coordinate axes built here are primed via a new util.with_fingerprint — ten sites across eight modules. The digest is cached on the axis and pickles with it, so computing it at construction spares the first consumer in every receiving process from recomputing it on every message; unpickling builds a new axis object per message, so a cold axis is re-checksummed forever.

The dummy message in sklearn.fit is left alone — it drives _reset_state and is never emitted.

Testing

369 passed, up from 360. The 9 new tests are mutation-checked: dropping the priming fails 6, and reverting STREAMING_DIMS to ("time",) fails 3.

Known limitation

FlattenTransformer raises on a message that declares chunk_dim, because it renames wintime and the inner ezmsg-sigproc transformer doesn't carry the field across. That's fixed in sigproc, not yet released, so it resolves when that ships. The tests here use undeclared messages, which is what this package actually receives today.

Dependencies

ezmsg-baseproc>=1.12.0, and the direct ezmsg pin raised to 3.10.0b2 alongside it — uv only enables pre-releases for a package named with a pre-release marker in this file, so leaving it to the transitive requirement fails resolution outright.

🤖 Generated with Claude Code

ezmsg-baseproc 1.12.0's default `_hash_message` folds in the message key, the
dims, the length of every dimension except the chunk dimension, the coordinate
*values* on those dimensions, and the gain and offset of any linear axis among
them. Four of the five overrides here were subsets of that and are gone.

Checked empirically rather than by reading, against a `(win, time, ch)` stream
where each step changes exactly one property:

  adaptive_decomp   own [0,1,2,6,7]  default [0,3,5,6,7]
  flatten           own [0,1,2,5,6]  default [0,3,5,6,7]
  ssr               own [0,6,7]      default [0,3,5,6,7]
  mlp_old           key + n_ch only, a strict subset

  (0 first  1,2 win-count jitter  3 relabel  5 window length  6 channel count
   7 new key)

Two of them were resetting on *win-count jitter* -- the chunk dimension's
extent, which is just how much arrived -- while missing a channel relabel.
`adaptive_decomp` had it exactly backwards: `settings.axis` defaults to
`"!time"`, so on a windowed message its `iter_axis` resolved to `time` and it
excluded the window length while including the window count. Every one of the
four now notices a relabel, which none of them did.

`ssr` also drops `group_spec_fingerprint`, whose O(1) "bank field present"
boolean was a deliberate concession to avoid a per-message cost that scaled
with channel count. `CoordinateAxis.fingerprint` removes that cost by
computing the digest once per axis object rather than once per consumer, so
the concession is no longer worth making. Its test asserted the old behaviour
by name and now asserts the new.

Two overrides stay, for reasons the default cannot cover:

* `adaptive_linear_regressor` has nothing to reset, so a constant is correct
  and cheapest. Changed from -1 to 0: -1 is the sentinel `_hash` starts at and
  that `_request_reset()` writes, so returning it made an explicitly requested
  reset compare equal and be swallowed.
* `sgd` needs a constant for a different reason, and this one was a live
  regression. Its `partial_fit` sets `_hash = 0` itself; training samples
  arrive as `(time, ch, freq)` and inference windows as `(win, time, ch, freq)`,
  so any layout-derived hash differs between the two and every alternation ran
  `_reset_state`, which calls `_refreshed_model()` and throws the fitted model
  away. `test_sgd` caught it. The assumption was previously inherited from
  baseproc's old constant default and is now stated where it is relied on.

`FlattenTransformer` gains `STREAMING_DIMS = ("win",)`. Its canonical input is
`(win, time, ch)`, where `win` grows per message and `time` is the lag
dimension whose length sizes the lag axis; the base class's `("time",)`
fallback excludes exactly the wrong one. This is load-bearing rather than
cosmetic, because released ezmsg-sigproc does not yet set `chunk_dim`, so
these messages arrive undeclared.

Coordinate axes built here are now primed via a new `util.with_fingerprint`:
the digest is cached on the axis and pickles with it, so computing it at
construction spares the first consumer in every receiving process from
recomputing it on every message. Ten sites across eight modules. The dummy
message in `sklearn.fit` is left alone -- it drives `_reset_state` and is never
emitted.

Requires ezmsg-baseproc 1.12.0. The direct `ezmsg` pin is raised to 3.10.0b2
alongside it: uv only enables pre-releases for a package named with a
pre-release marker in *this* file, so leaving it to baseproc's transitive
requirement fails resolution outright.

369 passed, up from 360. The 9 new tests are mutation-checked: dropping the
priming fails 6, and reverting STREAMING_DIMS to ("time",) fails 3.
@cboulay
cboulay merged commit 88d1f58 into dev Sep 4, 2026
9 checks passed
@cboulay
cboulay deleted the cboulay/baseproc-hashing branch September 4, 2026 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant