Declare chunk_dim and prime the channel fingerprint - #41
Merged
Conversation
Two things every consumer of this inlet needs, and only the inlet can supply. `chunk_dim` names the dimension messages append along. A consumer caches state against the stream's configuration -- channel count, labels, sample rate -- and must exclude the one dimension whose length is just however many samples arrived. It cannot reliably infer which that is: it is `time` here but `win` downstream of a windowing stage, so a guess either thrashes on chunk-size jitter or stops noticing real changes. It is `time` for a regular stream, whose LinearAxis offset advances per message, and equally `time` for an irregular one carrying per-sample timestamps in a CoordinateAxis. `CoordinateAxis.fingerprint` is a content digest, computed on first access and cached on the instance. The template builds its `ch` axis once and every message reuses that object, so priming costs one checksum per connection. Left cold it is computed by the first stateful consumer in this process -- and, because unpickling builds a new axis object per message, by the first consumer in every other process, on every message, until the inlet reconnects. Requires ezmsg 3.10.0b2 for both fields.
`pull_request` only triggered on PRs to main, but PRs here are based on dev -- every merged PR in this repo has dev as its base. The result is that a PR reports only the publish workflow's build job and none of the tests, which is how the previous commit reached review with a green tick and an unrun suite. blackrock, neo and nwb already list both branches; this matches them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things every consumer of this inlet needs, and only the inlet can supply. Both are set once per connection; neither costs anything per message.
chunk_dimA stateful consumer caches state — filter coefficients, per-channel history, resolved channel indices — against the stream's configuration: channel count, labels, sample rate. It has to exclude the one dimension whose length is just however many samples arrived.
It cannot reliably infer which that is. It's
timehere, butwindownstream of a windowing stage, so a consumer that assumestimeeither thrashes on chunk-size jitter or, worse, stops noticing a real change:Only the producer, which named the dims, knows. It is
timefor a regular stream, whoseLinearAxisoffset advances per message, and equallytimefor an irregular one carrying per-sample timestamps in aCoordinateAxis— in both cases the extent is just what arrived. The emit path already goes throughreplace(msg_template, data=..., axes={**msg_template.axes, "time": out_time_ax}), sofast_replacecarries the field for free.CoordinateAxis.fingerprintA content digest, computed on first access and cached on the instance — it's what lets a consumer notice that channels were relabelled at a fixed channel count, which is otherwise silent and numerically destructive:
The template builds its
chaxis once and every message reuses that object, so priming costs one checksum per connection. Left cold it's computed by the first stateful consumer in this process — and, because unpickling builds a new axis object per message, by the first consumer in every other process, on every message, until the inlet reconnects.Testing
49 passed. Six new tests, parametrised over regular and irregular streams, cover the template declaring its chunk dim, the channel axis carrying its fingerprint, both surviving a pickle round trip, and a reconnect re-priming for a new channel set rather than inheriting the old digest.
Dependency
ezmsg>=3.10.0b2for both fields. This is a pre-release pin until 3.10.0 final ships.🤖 Generated with Claude Code