Skip to content

feat: read a Stream in Workflow code, with Direct reads, appends and closes in Core - #2132

Draft
moetemp wants to merge 57 commits into
moe/streams-apifrom
moe/streams-direct
Draft

moetemp wants to merge 57 commits into
moe/streams-apifrom
moe/streams-direct

Conversation

@moetemp

@moetemp moetemp commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What changed?

This PR lets Workflow code read a Stream, with the same read, iter_read and iter_read_items as a client. It also makes the Workflow append and close Direct calls. Core calls the Store itself, and Workflow code gets each Record as it arrives.

It's a prototype, created for the purpose of getting your feedback on the API. Its base is #2129, and Core's side is temporalio/sdk-rust#1801.

events = workflow.get_stream(events_ref, body_type=ModelEvent)
status = workflow.get_stream(status_ref, body_type=str)

# Workflow code decides on each event while the model Activity still runs.
tools = []
async for record in events.iter_read_items(wait=timedelta(seconds=30)):
    event = record.body
    if event.kind == "tool_call":
        tools.append(
            workflow.start_activity(
                run_tool, event.tool_call, start_to_close_timeout=timedelta(minutes=1)
            )
        )
        await status.append(f"Running {event.tool_call.name}")
results = await asyncio.gather(*tools)
  • The read. read gives one page, and iter_read and iter_read_items give every page and every Record, as on a client. With wait, a call waits that long for a new Record and then gives an empty page. With no wait, it gives what is in the Stream. The iterators end when the Stream is closed and read to its end, and with no wait also when they catch up.
  • One Core read for a loop. Core keeps one long read open while the loop asks for its next page in the same activation. A read that no call waits on is closed before the activation completes. So it never holds a Workflow Task open, and its Records stay for the next call.
  • append. It keeps its signature. Core writes the Records with the Workflow as the Producer and the Workflow Task as the epoch, and no Local Activity runs.
  • close. It keeps its signature. The Workflow Task that asks for the close records only the intent. Core closes the Stream after the server took that task, and the next task gives the result. So an awaited close spans two Workflow Tasks, and no Activity runs. A Workflow Task that fails closes nothing, so its retry can still append.
  • History. One iteration costs one core_stream marker for each Workflow Task that it spans, whatever the number of Records. By default the marker has a range, a count and a digest. With workflow_read_replay_from=StreamReplaySource.HISTORY on the StreamConfig, it has the Records.
  • Replay. Workflow code gets the same Records at the same points. By default Core reads them from the Store again, so they must stay there for as long as the Workflow can replay. When Core can't read them, the Workflow Task fails with a StreamReplayError message.
  • Read-ahead. A read holds at most 4 MiB of its Records in the Worker while Workflow code is busy, then waits until the code takes them. StreamConfig.workflow_read_ahead_bytes changes that bound.
  • Batch window. A read gives Workflow code a batch at most once in 50 ms, so a busy Stream doesn't activate the Workflow for each Record. StreamConfig.workflow_read_batch_window changes it, and zero turns it off.
  • Lost Records. When a replay can't read its Records from the Store again, the Workflow Task fails. With StreamReplayError in workflow_failure_exception_types, the Workflow fails instead, as for NondeterminismError.
  • Metrics. Core records metrics for these calls, such as temporal_stream_read_records and temporal_stream_append_latency, each with a stream_store attribute. The README of the Streams helpers has the list.
  • Interceptors. A read starts as the system Nexus operation ReadRecords, so a Workflow outbound interceptor sees each page read, as it sees an append.
  • Stores. A Direct call needs a Store that runs on the bridge, as the Redis Store and MemoryStreamStore do. For a Store written in Python, append and close keep their Local Activity, and a read fails the Workflow.

It depends on two things that have no release yet.

Why?

A library that iterates a Stream in Workflow code needs each Record while its Producer still runs. ADK is an example, since its own loop reads the model through an async generator. On #2129, Workflow code can't read a Stream at all.

It has these costs.

  • A read keeps its Workflow Task open while a call waits on it. So a Signal, an Update, a cancel or an Activity result reaches the Workflow at the next Workflow Task, which is at the heartbeat at the latest (about 8 s by default).
  • A read that waits on a quiet Stream costs one Workflow Task about every 8 s. With wait, a quiet Stream gives empty pages, so code on a Stream that can be quiet for long uses iter_read and leaves the loop after an empty page.
  • An append is at-least-once. After a Worker crash, a rerun can write a second copy.
  • A replay on a Worker with no cache reads the Store again on every Workflow Task.
  • A close that code doesn't await can be lost, if the Worker dies when the Workflow ends in the task that asked for it. A graceful shutdown of the Worker waits for it.
  • A read of 200 Records that arrive over 2 s is 3 Workflow Tasks and 1 marker. The marker is 269 bytes, or about 35 kB with HISTORY.

This point is open.

  • A loop with no wait makes one more read at its end, which gives the empty page that ends it.

How did you test it?

The tests are in tests/nexus/test_stream_direct.py. They run on the local dev server with the memory Store. They cover a generator over iter_read_items that reads the Stream of a running Activity, a read with and with no wait, pages of max_records and max_bytes, a loop that keeps one Core read, a loop that awaits an append for each Record, a read across a quiet gap, the marker with and without Records, a replay with and without a Stream config, a Worker with no cache, a Workflow Task that fails in the middle of a read, a cancel and a continue_as_new with a read open, two loops from one offset, an interceptor that sees each page read, a payload codec, the metrics, the read-ahead bound, the batch window, a Workflow failure for lost Records, a tool Activity that starts from inside the loop, and a close in two Workflow Tasks (in a finally, after a failed task, and with a cancel). A Redis test reads with iter_read_items while an Activity appends, and replays its History from Redis.

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

@moetemp moetemp changed the title feat: add iter to the Workflow Stream, with Direct reads and appends in Core feat: add iter to the Workflow Stream, with Direct reads, appends and closes in Core Nov 15, 2026
@moetemp
moetemp force-pushed the moe/streams-direct branch 2 times, most recently from 65fd4dd to 88cddf9 Compare November 16, 2026 20:45
Core now reads and appends a Stream inside the Workflow Task. The bridge protos and the
payload visitor follow its new commands and jobs, so a payload codec covers their bodies.
The WIT declares `iter` as a helper method of the Workflow Stream. Only that file differs
from the pin before, so the bridge needs no rebuild.
The Stream WIT marks `iter` with `@nexus.helper sync=true` and returns a support class. Only
a fork of nexgen knows these tags until its `main` has them.
Core reads and appends a Stream of Workflow code through the runtime of its Store. The Worker
and the Replayer pass one for each Store of the client's Stream config that has one.
Core reads the Store and gives each page to the read in an activation, so a read costs one
marker for each Workflow Task. `StreamConfig.workflow_read_replay_from` decides whether
History holds the Records.
An append of a Workflow Producer on a Store with a Rust runtime goes to Core, which writes it as
the Workflow with the task as its epoch. Its tests now expect that Producer and that marker.
A marker with the Records needs no Store, so a Replayer with no Stream config can replay it.
The read takes its Store from the ref, as the first run had it.
The tests cover each stop, the marker, replays, a Worker with no cache, a failed task, a
Signal, a Store written in Python and a payload codec, and measure a read of 200 Records.
A read keeps its Workflow Task open, and a replay of a `STORE` read needs the Records in the
Store. The README, the changelog and the docs of the Store and the Replayer now say so.
Core now orders a Workflow's close after its own appends and evicts a replay whose marker
Records can't be read. Its protos didn't change, so the bridge protos and the visitor stay.
Core ends such a read with Done, and counts the idle time only while a read waits for the
Store. The stops, `iter_stream` and the README now say so.
A failed replay now reaches the caller with no cut-off, raised or in the result. New tests
cover a second read after a `break`, a completion, a cancel and a continue-as-new.
Core now gives a replayed call with no Store name the Store of its entry in History. So a
Replayer with no Stream config replays a Direct append and a `HISTORY` read of such a ref.
A second task that waited on one read replaced the future of the first, which then never
woke. A Record that failed to decode moved `last_offset` past it, so a new read lost it.
The Local Activity path, a client and an Activity run it with no serialization context. A
codec with a context then wrote bodies that no other reader of the Stream could decode.
Three Workflows dropped the outer loop that only the 1 s default needed. A plain `iter` now
reads across a quiet gap, and a zero ends with `Idle` on an empty Stream.
A read with a short timeout gave no Record when its Workflow Task started late, which failed
a test on the cloud run. The stubs of the Core protos have the new comments of the read bounds.
Core now records metrics for the Direct calls, and a graceful shutdown waits for a close in
flight. Its protos changed only in a comment that the generated files already had.
A close that the code doesn't await is lost only when the Worker dies or the Store doesn't
answer in 10 s. The README lists the metrics that Core records for the Direct calls.
One Workflow reads, appends and closes, and the buffer of its Runtime has each metric with
its count and Store. A shutdown returns after a close that the code didn't await.
The fork has `@nexus.helper`, its `sync` form and `@nexus.support-type` in one pull request.
The generated Stream package is the same with it.
The marker now holds each Stream ref once, so the generated stubs of `external_data` change.
No Python code reads a marker entry's Stream ref.
A read stops asking the Store while that many bytes of its Records wait for Workflow code.
The Worker and the Replayer give the bound to Core through the bridge.
A copy of each Record to another Stream is work that an Activity could do itself. The example
now starts each tool as its call arrives, and the docs tell of the read-ahead bound.
The first tool starts before the model Activity completes. A bound below 1 raises, and a
bound of 1 byte still gives every Record in order.
The stub of the Stream messages has the new comment, which says who sets the epoch.
Core adds a batch window and an idle default of 1 minute to a read, and can fail the Workflow
when a replay lost its Records. Only comments of the generated stubs change.
`StreamConfig.workflow_read_batch_window` bounds how often a read activates a Workflow.
`StreamReplayError` in the failure types fails the Workflow when a replay lost its Records.
A read with no `idle_timeout` now ends after 1 minute, so code on a quiet Stream checks
`done`. The docs also say that an append is at-least-once, and give the 1 MiB marker rule.
A window of 1 s joins the four later Records of a read in one batch, and a zero window gives
more batches. Lost Records fail the task by default, or the Workflow with the setting.
Core vendors the WIT that has no separate Workflow read, and its Store option is called
`producer_idle_ttl`, as on the base branch.
The base branch runs a Workflow's close on a Store written in Python in a Local Activity, so
the docs of this branch follow it.
A close on a Python Store runs in a Local Activity. The Direct close test of a failed task
needs a Workflow that fails in the task of the intent.
The Workflow `Stream` reads with the same `read`, `iter_read` and `iter_read_items` as a client,
so the generated Workflow target has no `iter`.
Workflow code reads with the same `read`, `iter_read` and `iter_read_items` as a client, so the
separate `iter` and `StreamReading` go. A Core read that no call waits on closes before the
activation completes, so it never holds a Workflow Task open for nobody.
The stops, `last_offset` of a read and the idle default of one minute are gone. A loop on a
quiet Stream ends with `iter_read` after an empty page, and the costs that still hold stay.
mypy wants a type for the handle of a Nexus read and a mapping for the read cases, and
basedpyright can't see that the instance uses `_DirectRead`.
@moetemp moetemp changed the title feat: add iter to the Workflow Stream, with Direct reads, appends and closes in Core feat: read a Stream in Workflow code, with Direct reads, appends and closes in Core Nov 17, 2026
A Workflow counts how often it wakes while it reads, and starts an Activity at a count. A
replay starts it at the same Record, since the marker has the activation of each page. With
other activations in the marker, the replay is a non-determinism error.
The earlier test counted the wakes with one condition. This one has the shape of the example, where each Record starts a task that counts and the loop starts an Activity when the count is 2.
An Activity between Records of one page is a later Workflow Task. The marker
stays on the task that received the page.
Core tries a Direct read, append or close again while its Store is unavailable.
A replay whose Store stays down fails only its Workflow Task.
A proxy in front of Redis drops each connection while it is down. The tests check that an
append and a read give each Record once, and that a replay fails only its task.
Core tries a Direct call again while its Store is unavailable. A replay whose Store stays down
fails only its Workflow Task, so `StreamReplayError` says it is not that error.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant