Skip to content

Declare chunk_dim and prime the channel fingerprint - #41

Merged
cboulay merged 2 commits into
devfrom
cboulay/chunk-dim-and-fingerprint
Sep 4, 2026
Merged

Declare chunk_dim and prime the channel fingerprint#41
cboulay merged 2 commits into
devfrom
cboulay/chunk-dim-and-fingerprint

Conversation

@cboulay

@cboulay cboulay commented Sep 3, 2026

Copy link
Copy Markdown
Member

Two things every consumer of this inlet needs, and only the inlet can supply. Both are set once per connection; neither costs anything per message.

chunk_dim

A stateful consumer caches state — filter coefficients, per-channel history, resolved channel indices — against the stream's configuration: channel count, labels, sample rate. It has to exclude the one dimension whose length is just however many samples arrived.

It cannot reliably infer which that is. It's time here, but win downstream of a windowing stage, so a consumer that assumes time either thrashes on chunk-size jitter or, worse, stops noticing a real change:

post-Window (win, time, ch)   0=first  1,2=win-count jitter  3=relabel  4=same  5=window-len change
  consumer assumes "time":  resets at [0, 1, 2, 3]
  message declares "win":   resets at [0, 3, 5]      correct

Only the producer, which named the dims, knows. It is time for a regular stream, whose LinearAxis offset advances per message, and equally time for an irregular one carrying per-sample timestamps in a CoordinateAxis — in both cases the extent is just what arrived. The emit path already goes through replace(msg_template, data=..., axes={**msg_template.axes, "time": out_time_ax}), so fast_replace carries the field for free.

CoordinateAxis.fingerprint

A content digest, computed on first access and cached on the instance — it's what lets a consumer notice that channels were relabelled at a fixed channel count, which is otherwise silent and numerically destructive:

first 4 samples of the new channel 'armB-1':
  filter state carried from armA: [-5.919  -4.772  -0.5114  1.064]
  with a correct reset:           [ 0.     -0.0013 -0.0023 -0.0026]
  max |difference| = 11.12   vs new-data amplitude 0.024

The template builds its ch axis once and every message reuses that object, so priming costs one checksum per connection. Left cold it's computed by the first stateful consumer in this process — and, because unpickling builds a new axis object per message, by the first consumer in every other process, on every message, until the inlet reconnects.

Testing

49 passed. Six new tests, parametrised over regular and irregular streams, cover the template declaring its chunk dim, the channel axis carrying its fingerprint, both surviving a pickle round trip, and a reconnect re-priming for a new channel set rather than inheriting the old digest.

Dependency

ezmsg>=3.10.0b2 for both fields. This is a pre-release pin until 3.10.0 final ships.

🤖 Generated with Claude Code

Two things every consumer of this inlet needs, and only the inlet can supply.

`chunk_dim` names the dimension messages append along. A consumer caches state
against the stream's configuration -- channel count, labels, sample rate -- and
must exclude the one dimension whose length is just however many samples
arrived. It cannot reliably infer which that is: it is `time` here but `win`
downstream of a windowing stage, so a guess either thrashes on chunk-size
jitter or stops noticing real changes. It is `time` for a regular stream, whose
LinearAxis offset advances per message, and equally `time` for an irregular one
carrying per-sample timestamps in a CoordinateAxis.

`CoordinateAxis.fingerprint` is a content digest, computed on first access and
cached on the instance. The template builds its `ch` axis once and every
message reuses that object, so priming costs one checksum per connection. Left
cold it is computed by the first stateful consumer in this process -- and,
because unpickling builds a new axis object per message, by the first consumer
in every other process, on every message, until the inlet reconnects.

Requires ezmsg 3.10.0b2 for both fields.
`pull_request` only triggered on PRs to main, but PRs here are based on dev --
every merged PR in this repo has dev as its base. The result is that a PR
reports only the publish workflow's build job and none of the tests, which is
how the previous commit reached review with a green tick and an unrun suite.

blackrock, neo and nwb already list both branches; this matches them.
@cboulay
cboulay merged commit c92b128 into dev Sep 4, 2026
14 checks passed
@cboulay
cboulay deleted the cboulay/chunk-dim-and-fingerprint branch September 4, 2026 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant