Skip to content

Commit 116f765

Browse files
committed
Added a pipeline that runs a chain of Stream steps as one call.
The take, filter and reduce calls don't chain. The pipeline compiles a chain into one of them, so in Workflow code it stays one Local Activity and one marker.
1 parent f81fa77 commit 116f765

5 files changed

Lines changed: 949 additions & 4 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add `temporalio.contrib.streams.pipeline`, which chains `filter`, `map`, `take_until` and `limit` steps over a Stream and runs the whole chain as one `take`, `filter` or `reduce` call. In Workflow code a chain is one Local Activity and one marker in History.

‎temporalio/contrib/streams/README.md‎

Lines changed: 42 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22

33
> This package is experimental. Its API can change in a later release.
44
5-
`temporalio.contrib.streams` holds helpers that sit on top of the Streams API of the SDK. A
6-
helper takes a client `Stream` or a Workflow `Stream`.
5+
`temporalio.contrib.streams` holds two helpers that sit on top of the Streams API of the SDK.
6+
Each one takes a client `Stream` or a Workflow `Stream`, and works the same in client, Activity
7+
and Workflow code.
78

89
## `BufferedAppender`
910

@@ -21,3 +22,42 @@ async with BufferedAppender(tokens) as appender:
2122
```
2223

2324
The end of the `async with` block writes the Records that are still in the buffer.
25+
26+
## `pipeline`
27+
28+
`pipeline` chains steps over a Stream: `filter`, `map`, `take_until` and `limit`. A step runs
29+
nothing. The end of the chain runs the whole chain as one `take`, `filter` or `reduce` call of
30+
the Stream, so in Workflow code the chain is one Local Activity and one marker in History.
31+
32+
```python
33+
from datetime import timedelta
34+
35+
from temporalio.contrib.streams import pipeline
36+
37+
text = await (
38+
pipeline(tokens)
39+
.filter(lambda record: record.body.kind == "text")
40+
.map(lambda record: record.body.text)
41+
.take_until(lambda text: text.endswith("\n"))
42+
.reduce(lambda joined, text: joined + text, "", timeout=timedelta(minutes=1))
43+
)
44+
```
45+
46+
An item is a `StreamRecord` until the first `map`, and the value of that map after it. The
47+
functions of the steps must keep no state, since a Local Activity can run them again.
48+
49+
Each end takes `timeout`, `from_offset`, `idle_timeout`, `max_records` and `max_bytes`, as the
50+
three calls do. Each result has `last_offset`, `stop` and `done`. `last_offset` is where the
51+
scan stopped, so a loop that gives it as the next `from_offset` loses and repeats no Record.
52+
53+
| End | Call of the Stream | Gives back | Goes into History in Workflow code |
54+
| --- | --- | --- | --- |
55+
| `records(...)` | one `filter`, or one `take` when the chain keeps every Record | `StreamRead[BodyT]` with the kept Records | the Records |
56+
| `reduce(function, initial, ...)` | one `reduce` | `StreamReduce[ValueT]` with the folded value | the value |
57+
| `collect(...)` | one `reduce`, or as `records` when the chain has no `map` | `StreamReduce[list[T]]` with the items | the list |
58+
| `count(...)` | one `reduce` | `StreamReduce[int]` with the number of items | the count |
59+
60+
`records` runs only a chain of `filter` steps, with at most one `take_until` and at most one
61+
`limit` as its last step. For any other chain it raises `TypeError`, and `collect` or
62+
`reduce` runs it. In Workflow code a result must fit in 1 MiB once encoded, since it goes
63+
into History.
Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,18 @@
11
"""Helpers that sit on top of the Streams API.
22
3-
:py:class:`BufferedAppender` batches the appends of one Stream. It takes a client Stream or a
4-
Workflow Stream.
3+
:py:class:`BufferedAppender` batches the appends of one Stream. :py:func:`pipeline` chains
4+
steps over a Stream and runs the whole chain as one ``take``, ``filter`` or ``reduce`` call.
5+
Both take a client Stream or a Workflow Stream.
56
67
.. warning::
78
This package is experimental and may change in future versions.
89
"""
910

1011
from temporalio.contrib.streams._appender import BufferedAppender
12+
from temporalio.contrib.streams._pipeline import Pipeline, pipeline
1113

1214
__all__ = [
1315
"BufferedAppender",
16+
"Pipeline",
17+
"pipeline",
1418
]

0 commit comments

Comments
 (0)