Replay service cooperative cancellation at the canonical boundary - #90
Merged
Merged
Conversation
Member
Author
|
All source checks have passed for The connected cooperative worker/client path is still the next implementation step. Existing integration success verifies the preserved supported protocol 1.19 surface. It does not qualify the new delivery intent, heartbeat observation, acknowledgment recovery or active local activity fencing against Server PR291. This PR remains a draft and advertises no cooperative cancellation capability. The completed local test worktree and tool images are removed, with the active branch preserved. |
Member
Author
Request and delivery client source
Delivery requires explicit compatible protocol 1.20, worker credentials, the recorded lease owner/attempt and a canonical call/range. Success must acknowledge the exact task/request/boundary. Transport retry preserves that body. Lease and attempt refusals retain Server's machine reason and do not retry as another claim. Local source check on Python 3.12.14: pytest tests/test_cooperative_cancellation_client.py tests/test_client.py tests/test_cooperative_cancellation.py tests/test_replay_regression_corpus.py -q
ruff check src/ tests/
mypy src/durable_workflow/
python scripts/ci/validate-regression-corpus.py --base-ref f542486e125fdb2738d0d6185ff8fe748e89ddcdAll 323 tests pass, including 52 new carrier cases. Lint, strict typing and corpus policy pass. CI36793706333 is qualifying this head. The next Worker change must consume claim and heartbeat observations, persist the replay delivery intent before throwing, then reload Server-issued canonical history with the same owner/attempt. Active local activity results must be fenced across delivery, death and reclaim. Default protocol/capability advertisement remains unchanged until that path, all three SDK implementations and the exact published tuple qualify. |
Member
Author
Client source qualification completeAt The new client carrier requires explicit compatible capability discovery for cooperative requests. A run-bound handle preserves its run fence. Worker delivery requires protocol 1.20, worker credentials, claim owner/attempt and the authored call range. Duplicate requests preserve Server's original identity and deadline. Lost acknowledgments retry the identical delivery, and malformed or mismatched acknowledgments are rejected. Earlier immediate cancellation and termination remain covered. The completed source worktree, dependencies, caches, tool images and build context have been removed. The active draft branch remains. Next is Worker integration for claim/heartbeat observations, canonical history refresh, delivery persistence and active local-result fencing, then the accepted connected failure cases and equivalent PHP/Rust behavior. Default protocol and published capability claims remain unchanged. |
Member
Author
Worker source gates completedExact head The local suite passed 1,607 tests with two existing skips and 31 integration cases deselected. Thirty Worker cases exercise claim/heartbeat observation, canonical paging and delivery, earlier commands, acknowledgment loss, local work and lease fencing, shielded cleanup, cold replay and incapable registrations. The draft remains protocol 1.19 by default. Connected candidate Server qualification, actual process death/reclaim, shutdown during shielded cleanup, synchronous and remote in-flight work, deadlines and termination remain required, along with PHP/Rust equivalents and coordinated publication. Existing Server integration covers the preserved current protocol surface. Cleanup is complete: the clean source worktree, virtual environment, test logs, build context and both disposable tool images are removed. The remote branch and this evidence retain the implementation and reproducible commands. |
Member
Author
Negotiated local cleanup shutdown and synchronous executionExact head Synchronous local handlers execute off the event loop in a lazy pool bounded by Local evidence: 1,611 tests passed, two existing skips, 31 integration cases deselected, 26.77seconds. All 34 focused Worker cases pass. Ruff, strict mypy for all 26 source modules and the revision-aware corpus guard pass. The synchronous observation case deliberately limits replay to one thread to expose a shared-pool deadlock. Replacement source cases preserve canonical request/history and use a different lease owner and attempt5 without redelivery. Execution uses Python3.12.14, UID/GID1000:1000, two CPUs, 1GiB memory with no additional swap allowance and pids256. Python base index is ruff check src/ tests/
mypy src/durable_workflow/
pytest tests/ -m "not integration" -q
python scripts/ci/validate-regression-corpus.py --base-ref f542486e125fdb2738d0d6185ff8fe748e89ddcdNormal CI for the exact head completed successfully at 01:25:59 UTC on October 1. All jobs pass, including Python 3.10/3.11/3.12, existing Server integration, lint, corpus, package and target-branch qualification. Docs and public boundary checks also pass at the same head. The draft remains pending connected candidate Server qualification, actual native process replacement, native shutdown/in-flight work, deadline/termination and acknowledgment-loss cases, PHP/Rust equivalents and coordinated publication. Default protocol1.19 and current registration advertisement remain intact. Completed local worktree, dependencies, build context, diagnostics and task images have been removed. No published support claim is added by these source checks. |
Member
Author
Connected Python/Server source qualificationAt October 1, 2026, 02:25 UTC, Python
The host used isolated containers at UID/GID 1000, 2 CPU and 1 GiB memory with no additional swap per tool/service. Runtime base is published Server 2.4.34 index Local checks:
Normal Python CI and the explicit Python/MySQL candidate run pass every job at the exact head. The candidate integration completed at 02:29:45 UTC, with 33 passed and one existing CLI skip in 80.19 seconds. All eleven cooperative cases are present and passed in the downloaded JUnit artifact, which expires October 8. The log verifies the exact Server SHA above, and stack teardown passed. Supported Python 3.10/3.11/3.12, lint, corpus, package, docs and public boundaries pass. Server source/corpus CI passes 2,279 tests and 47,800 assertions. MySQL replay/query topology passes. Polling bounded-growth smoke and its performance qualification also pass at this exact head. All normal Server gates are complete. PHP/Rust equivalents, coordinated specification/capability activation and exact published tuple conformance remain required. Both PRs remain drafts. Local stack and disposable source/dependency/proof/tool resources are removed. Downloaded CI diagnostics and transport files are removed after updating this record. |
This was referenced Oct 1, 2026
Merged
Member
Author
|
Next source phase: supervise actual cooperative remote activity callbacks through the qualified Server readonly observation route (Server073a516bbd4063f57d4ad65ad732ec423131c8ff). Keep worker registration and only authored activity progress, observe canonical cancellation/attempt/session/deadline fences, abandon on failed observation or shutdown expiry, and reject late heartbeat/result/failure publication. Qualify blocked async callbacks and synchronous handlers while preserving event-loop responsiveness and the existing bounded thread-pool model. Python threads cannot be forcibly stopped, so prove their late publication is fenced and document the remaining external-effect/process-supervisor responsibility. Extend the connected actual-worker cases and retain exact gates before claiming parity. Ordinary protocol1.19 registration and published capabilities stay unchanged. |
Member
Author
|
Remote worker qualification is running at Python The exact-head ordinary CI and connected integration pass. The opt-in candidate run 36825588696 passes 39 connected cases, fails the new killed-remote-owner recovery case, and retains the existing CLI skip. All six new async/sync blocked-callback and shutdown cases pass. The replacement process in the kill case times out before receiving its workflow claim. Its result is not a qualified recovery claim. Next action: reproduce that case in an isolated local stack using the same Server commit and inspect task leases, registration state and durable history. Determine whether the failure is a fixture bound or a recovery defect before changing it, then rerun the exact-head candidate gate. The failed runner completed teardown successfully and retained JUnit in artifact |
Member
Author
|
The isolated reproduction identifies the failure as Workflow #599. A still-issued workflow poll claims the new task for the killed process. Native repair then hides that expired workflow lease behind the older activity lease and repeatedly returns A native regression using the actual workflow and activity bridges fails on the published baseline, while its fresh-lease case passes. Prioritizing expired leases in the native run summary makes both cases pass, with 33 assertions checking that only the workflow claim changes and the activity task, execution, attempt and history stay unchanged. The unchanged Python SIGKILL case passes in 18.28 seconds when the isolated Server loads that corrected source. Its 30-second claim bound is unchanged. The replacement workflow task has attempt 2 and repair count 1, commits canonical delivery/cleanup, and the dead activity owner's late completion and failure are rejected. This local source result is the counterfactual for the defect. The full cooperative source suite, Workflow quality cycle, patch publication and exact dependent candidate gate are still running or pending. Protocol 1.20 remains opt-in. Next action: qualify and publish the Workflow repair patch, update the Server candidate to that exact package, then rerun Python's entire connected candidate suite and retire the local qualification resources. |
Member
Author
|
The cooperative Server candidate now consumes published Workflow 2.3.1 at Workflow's complete source matrix and all 16 published Laravel/PHP upgrade combinations pass. Server #292 is separately qualifying stable image 2.4.35. A focused ordinary-protocol signal drill reproduces the expired-workflow/older-active-activity defect on published Server 2.4.34 and PHP SDK 2.1.6, so this repair also affects existing service callers. The entire Python candidate CI is dispatched at unchanged Python |
Member
Author
Exact native correction requalification completePython head Server's current exact head passes feature/corpus with 2,300 tests and 47,995 assertions, MySQL HTTP topology, MySQL/Postgres rolling, bounded polling, chart validation/install and public boundaries. The correction is also published independently in stable Server 2.4.35 with its default protocol 1.19. Its focused published recovery test and all 12 published portable lifecycle cells pass. These candidate source results do not activate cooperative protocol 1.20. Next action is finish Rust's equivalent actual remote-worker path and active activity-attempt reclaim qualification, then coordinate publication and exact published tuple gates. PHP/Python/Server PRs remain drafts through that work. |
Member
Author
Requalified against the current Server and published Native packagePython The connected suite reports 40 passed, 1 skipped in 128.61 seconds, including all 18 cooperative cases. These include actual async/synchronous remote callbacks with and without authored heartbeats, accepted owner registration heartbeats, shutdown fencing, canonical waiting/delivery, lost replies, local callback result suppression, shielded cleanup replacement, real remote-owner SIGKILL, cleanup reclamation in a new process, and cleanup deadline/termination fencing. Ordinary smoke, payload, session and memo integration cases also pass. The supported Python matrix, package, corpus and Avro checks pass in the same run. Raw connected JUnit is retained for seven days. CI removed its isolated stack, network, payload volume and task images. The Server prefix successor fix is included in this candidate. Next: complete active remote-attempt replacement qualification and the remaining shared per-language scenarios, then qualify the exact published tuple. This is source qualification, and this PR remains a draft. Published protocol/capability defaults are unchanged. |
Member
Author
Child waits release the Python claimSource The worker returns to polling without publishing completion or failure and Focused local source checks passed: 151 tests, Ruff, and mypy over all 26 Connected child-policy qualification remains required. This is a source draft, |
Member
Author
Scoped cancellation origin source evidence, part 00 of 00–01Python Archive: |
Member
Author
Scoped cancellation origin source evidence, part 01 of 00–01Python Archive: |
Member
Author
|
Merged the independently qualified Python 2.3.9 HTTP-timeout fix into this draft. The merged production change uses the configured Client timeout for an omitted
The earlier unclaimed ready task's cause remains unresolved. A configured |
Member
Author
|
Refreshed exact-source qualification passed after incorporating the independently
The mixed PHP parent → Python child → Rust remote activity plus PHP local The real cleanup SIGKILL and replacement retain the same delivery boundary. Scope authoring still has 13 real history page requests, one side effect and a The Python timeout fix is delivered as ordinary protocol-1.19 SDK 2.3.9 in Next cancellation gate: implement and qualify scoped delivery and selective Raw connected artifacts are retained by the linked GitHub runs for 90 days. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the Python consumer for shared cancellation #136.
Supported release boundary
Whole-run cooperative requests, immutable cancellation context and inherited
deadline, Activity/Child TryCancel, WaitCancellationCompleted and Abandon,
heartbeat-independent async/process callback supervision, stale fencing and
shielded durable cleanup recovery. Independently cancellable scopes stay an
opt-in source preview, disabled by default and outside the supported RC claim.
Ordinary workers remain protocol 1.19. Cooperation explicitly selects 1.20.
Qualification
Connected source qualification
passes 67 cases with zero failures/errors and one existing CLI-unavailable skip.
Actual callback stop, original delivery/deadline and cleanup-worker SIGKILL
recovery pass. Runtime code is unchanged by the subsequent RC metadata/docs.
Current head
ce256a9406a3539b47f9ae329956f38aa4fb0523passes all ordinaryPython 3.10–3.12, package, lint, type and corpus gates.
The PHP/Python/Rust mixed case
passes 237 assertions, including replacement replay and both runs Cancelled
17.169470 seconds after the original request, before its 30-second deadline.
Publication
Candidate Python SDK 2.4.0-rc.1 (PyPI 2.4.0rc1) targets Server 2.5.0-rc.1.
Publish the exact reviewed source and verify registry identity. The final
published mixed scenario and stable qualification remain required before #136 closes.