Correct service-mode cancellation claims and add cooperative request parity

[rmcdaniel]

Problem

The stable service-mode POST /workflows/{id}/cancel route calls Workflow's terminal attemptCancel() and immediately closes the run, cancels open tasks/timers, and revokes leases. It is not the separate cooperative request path shipped for embedded Laravel in workflow#483 / Workflow 2.2.0. A real-Server Python integration case in sdk-python#69 confirms that an in-flight local activity's next heartbeat loses the lease and no local completion is committed.

Published Python/Rust SDK docstrings and the language-neutral docs call cancel_workflow cooperative/graceful and imply workflow cleanup runs. That claim is false for the current service-mode route and could cause customers to skip external cleanup or reconciliation.

Immediate correction

• Audit the PHP, Python, Rust SDK guides/API docs and main service-mode docs. State plainly that current cancel is terminal; terminate is also terminal with a distinct outcome/reason. Neither guarantees workflow-code finally cleanup. Do not change stable endpoint semantics under a documentation fix.
• Preserve the accurate embedded-Laravel cooperative-request guidance and clearly label its deployment mode.
• Verify rendered/public guidance after publication with focused link/content checks. Do not turn this into a site-wide screenshot pass.

Additive parity work

• Design a separate service-mode cooperative cancellation request using the already shipped Workflow engine primitive without repurposing /cancel. Expose the request/observable pending state in Server and PHP/Python/Rust SDKs only after one shared protocol/conformance contract is qualified.
• Cover request-before/after task claim, waiting runs, bounded cleanup, duplicate requests, worker loss/replay, termination during cleanup, SDK task cancel_requested delivery, and local/remote activity fencing. External effects remain at-least-once and need idempotency/reconciliation.
• Publish capability claims only after exact published Server + SDK artifacts pass cross-language conformance. Keep terminal cancel behavior and compatibility intact.

This is a confirmed product/docs contract gap, not a request to reopen the completed embedded feature. Own the immediate correction first; the additive protocol work follows current accepted release commitments. No private Cloud or customer data belongs here.

Closure requirement and cancellation model work

The operator requires this issue to remain open until Durable Workflow's shipped cancellation model is clearly stronger than competing workflow products. Portable compatibility and passing the existing qualification scenarios are prerequisites. They do not complete this issue.

Evaluate current Temporal, Restate and DBOS cancellation contracts and their strongest supported configurations using primary sources, exact versions and reproducible published-artifact scenarios. Document useful customer advantages, competing strengths and material gaps. Fix the gaps needed for a coherent model rather than treating a checklist or unsupported superiority claim as evidence.

Prioritize the following decisions and implementation using engineering judgment:

1. Verify parent-close RequestCancel semantics. Make cooperative propagation explicit and preserve original request lineage and immutable cleanup budget. Define deliberate migration/compatibility behavior for any existing terminal contract.
2. Define Activity and Child operation policies equivalent to request-and-continue, wait-for-cancellation-completion and abandon. Specify what acknowledgement proves, what can still run externally, and how cleanup avoids overlapping live work.
3. Make requestCancellation() the discoverable cooperative API, preserving the stable terminal cancel() behavior with clear naming and documentation.
4. Expose a coherent durable cancellation context and observable lifecycle: request identity, reason, requester/source, timestamps, original deadline and lineage, plus delivery, cleanup outcome, expiry and termination. Connect workflow authoring to API/CLI/UI diagnostics.
5. Qualify and document cancellation observation independent of application heartbeats in PHP/Python/Rust, including worker loss, attempt replacement, bounded cleanup and stale-authority fencing. Specify scheduling and transport limits precisely.

Nested scopes/shielding, inherited budgets across the parent/child/activity tree, deterministic remaining-time helpers, capability diagnostics, bulk requests and downstream fencing tokens will be assessed against those outcomes. Existing mechanisms should be reused where they satisfy the contract. Additional work belongs in the owning repositories and remains linked here.

The current recovery gates continue, but defer freezing protocol 1.20 or publishing the planned feature RC tuple until the model decisions are settled. Preserve protocol 1.19 and existing terminal routes. Publish stable capability claims only after the final exact Server/PHP/Python/Rust artifacts, specification, customer-facing diagnostics and comparative evidence meet these closure requirements.

Required mixed-language cascade before closure

Use one exact published artifact tuple in an isolated stack:

Parent PHP workflow
  +-- Python child workflow
  |     +-- Rust remote activity
  +-- PHP local activity

Request cancellation with one root identity and a cleanup deadline exactly 30 seconds after the original request. Demonstrate all of the following in the same run:

• The Python child receives a genuine cooperative request and runs cleanup.
• The Rust remote activity and PHP local activity stop without application heartbeats. Record actual callback stop observations and the SDK/runtime acknowledgements separately from loss of commit authority.
• The stale Rust activity attempt cannot publish a late result.
• SIGKILL a workflow worker during cleanup. A fresh replacement replays the original canonical delivery boundary and resumes durable cleanup.
• A duplicate request returns the original root identity and deadline, including after replacement. No hop, retry or recovery refreshes the budget.
• Cleanup and all runs converge to Cancelled before the original deadline. Record elapsed time, canonical histories and cleanup outcomes. An expired budget cannot substitute for successful recovery within this scenario.
• One UI/API view explains the entire cascade: root request, original deadline, parent/child lineage, activity stop and fencing, worker loss/replacement, cleanup progress and final outcomes.

Retain commands, artifact versions and digests, raw results, histories and the cascade view. This scenario is a required closure gate alongside the stronger model and competitive evidence above. Keep the issue open until it passes against artifacts users can install.

[rmcdaniel]

Immediate factual correction is in review:

• Main docs: Correct service-mode cancellation guidance durable-workflow.github.io#139
• PHP SDK: Clarify PHP service-mode cancellation outcome sdk-php#74
• Python SDK: Correct Python cancellation and cleanup docs sdk-python#70
• Rust SDK: Correct Rust cancellation API docs sdk-rust#48
• CLI: Clarify terminal workflow cancellation in CLI cli#32

Focused local verification passed: docs production build, Python 239 tests and Ruff, CLI 18 tests, and public-boundary scans. CI is running. The separate additive service-mode cooperative request remains open; none of these PRs changes the existing terminal route.

[rmcdaniel]

Immediate contract correction is merged: docs#139, sdk-php#74, sdk-python#70, sdk-rust#48, and cli#32. Main docs are live and show the terminal service-mode behavior. The CLI 2.1.1 patch release has been dispatched to ship corrected installed help/output; publication verification is pending. Service-mode cooperative request parity remains a separate additive protocol task in this issue.

[rmcdaniel]

Immediate factual correction is delivered: main docs#139, sdk-php#74, sdk-python#70, sdk-rust#48, and cli#32 merged; their public documentation pages were spot-checked after deployment. CLI 2.1.2 is published with the corrected installed workflow:cancel help and the repaired default upgrade path (cli#33 / PR#34; docs manifest PR#140). Published binary/checksum/installer/upgrade dry-run checks passed. The separate additive service-mode cooperative cancellation request remains open here; no existing /cancel behavior changed.

[rmcdaniel]

Service-mode implementation triage: Workflow already persists CooperativeCancellationRequested and a bounded cleanup deadline through WorkflowStub::attemptRequestCancellation(). Server currently exposes only terminal /cancel (and run-scoped terminal cancel); its workflow-task poll payload does not deliver the pending cancel_requested state that Python workers consume, so adding a request route alone would not make cleanup cooperative. The additive path needs a distinct request endpoint, durable pending-state delivery on claimed/replayed workflow tasks, deadline/fencing behavior across worker loss and termination, and one shared PHP/Python/Rust contract test. Preserve existing /cancel semantics. Next implementation should start at the Server request/poll boundary, then qualify the published Server + SDK tuple before advertising parity.

[rmcdaniel]

Portable implementation audit (2026-09-24): WorkflowStub::attemptRequestCancellation() already persists the request, deadline, history event and a ready task when no open task exists. Embedded WorkflowExecutor::run() checks that deadline and performs terminal cleanup. In contrast, Server WorkflowTaskPoller::taskPayload()/refreshCachedTaskPayload() do not project the pending cancellation flag or deadline, and the portable DefaultWorkflowTaskBridge has no cancellation-deadline path. PHP/Python/Rust workers already consume a task-level cancel_requested flag, but Server currently supplies it only for closed-run rejections, not an active cooperative request. A new HTTP route by itself would therefore be an incomplete and misleading feature, especially for a request arriving after task claim or a worker lost during cleanup. The first coherent implementation must couple the distinct request endpoint with durable task delivery (including cached/reclaimed leases) and a portable deadline/fencing transition; then test before/after claim, restart, duplicates, deadline expiry and terminal cancel/terminate races on one published tuple. Preserve existing /cancel semantics. No service-mode parity claim is made from this audit.

[rmcdaniel]

Immediate customer-facing contract correction is shipped. PHP SDK #74, Python SDK #70, Rust SDK #48, and main docs #139 already corrected the terminal service-mode cancel claim. I found one remaining false claim in the embedded child-workflow guide: ParentClosePolicy::RequestCancel actually invokes terminal attemptCancel(), not cooperative requestCancellation(). Docs #142 corrected it and removed a misleading Rust-guide link. Its checks and Pages deployment passed.

Focused live-content checks now show the terminal/no-cleanup contract on the PHP failures/retries page, Python workflow reference, Rust Client API, main Server API reference, and the corrected child-workflow page. These checks validate published wording, not service-mode cooperative behavior.

Leave this issue open for the additive service-mode cooperative-request contract and cross-language conformance; do not repurpose the stable /cancel endpoint. The current documented request_cancel parent-close policy remains terminal despite its name, which must be accounted for in any later API design.

[rmcdaniel]

Current implementation boundaries, September 30

Rechecked current source after the published operator-diagnosis fixes. This is an implementation audit, not a cooperative-cancellation runtime pass.

• Server 76ad11fc64a0fee365fe68c6b7c5f546353a52cc still exposes terminal /cancel. Workflow-task poll and cached-poll refresh do not project the durable cooperative request/deadline. The workflow-task heartbeat maps lease renewal/status without a pending cooperative notification.
• Workflow 79a7127e41a62c153800923967fa1d09433f70a2 already records one idempotent request and its original cleanup deadline. TaskWatchdog identifies an expired cancellation deadline and creates a ready task. Embedded WorkflowExecutor performs terminal cancellation at that boundary. Portable DefaultWorkflowTaskBridge::complete() checks the ordinary run timeout before commit but has no corresponding cooperative cleanup-deadline transition. The portable heartbeat path can renew an active run without exposing that request.
• Python f542486e125fdb2738d0d6185ff8fe748e89ddcd consumes a task's initial cancel_requested flag. Its local-activity lease renewal validates the renewed lease without copying a newly received request into the active task. Request-after-claim therefore needs an implemented delivery path, including local-activity boundaries.
• Rust c70c18b6d58657cc99b875ddaf14be8812367682 consumes the initial task flag. Its history-derived cancellation state recognizes WorkflowCancellationRequested / WorkflowCancelRequested, while the engine's canonical durable request event is CooperativeCancellationRequested. The shared replay contract must specify the actual marker and delivery boundary.

The first coherent implementation should establish one portable contract in Workflow and Server before enabling clients: a distinct request operation, original immutable deadline, observable pending state, delivery before and after task claim, cleanup/replay identity and runtime-owned terminal deadline enforcement. Reuse the existing watchdog and task lease fencing. Keep terminal /cancel, termination and parent-close behavior intact. Older workers must continue to fail closed for unsupported cooperation.

Next action: add the portable engine boundary with focused deadline/commit/worker-loss regressions, then couple Server's request endpoint, poll/cached delivery and heartbeat notification to it. PHP/Python/Rust implementations and one published-artifact conformance tuple follow that shared contract. A request route or capability flag alone does not complete this issue. The earlier published wording corrections remain valid.

[rmcdaniel]

The first engine boundary is now owned by durable-workflow/workflow#595. Implementing and reproducing late portable completion/heartbeat after the existing durable cleanup deadline before coupling the Server request/notification contract. The separate request endpoint and published cross-language capability qualification remain required here.

[rmcdaniel]

The engine prerequisite is published as Workflow 2.2.22, through durable-workflow/workflow#595 and PR #596. Source qualification and the exact published-package/Laravel upgrade matrix pass.

Portable heartbeat and command acceptance now share the engine's persisted cleanup deadline transition. Expiry revokes open work and records one original-request cancellation. Before expiry, valid leased cleanup authority remains. Server's pinned image follow-through is durable-workflow/server#286 and PR #287, with HTTP regressions and exact published-artifact qualification in progress.

The next contract work remains the service request operation, request visibility after an active claim, immutable deadline delivery, and shared PHP/Python/Rust deterministic cleanup/replay behavior. That needs a qualified worker-protocol evolution and fail-closed treatment for workers that cannot cooperate. No new service cooperative capability is advertised by this patch.

[rmcdaniel]

The engine deadline prerequisite and published consumers are complete:

• Workflow #595 is closed with published Workflow 2.2.22, source 774afd5d8951c28b48a6d6632854fd26ec73ffd5, focused/cold-reload coverage and full source/published-package qualification.
• Server #286 is closed with published Server 2.4.32 and exact multi-platform digest sha256:1bc35cc682b53d20eecefc9b9780310d6a4855f3fa69117646b50f7489d6332f. The actual published HTTP boundary checks preserve original request identity, refuse expired heartbeat/completion and cancel once. Published PHP/Python/Rust lifecycle passes; the reproducer and raw outcomes are retained on the release.
• Sample App #125 is closed after main embedded/polyglot tests and both published devcontainer architecture/startup checks. Qualified main image index in both registries: sha256:ab5a7389a97aa74ba8b25b89f387af9166bc8a3a4fb83a27c4bb8d4c17854fa5.

This issue stays open for the additive service-mode contract. Next: specify and implement a separate durable request operation with observable pending state, request/deadline delivery after an active claim, deterministic PHP/Python/Rust cleanup/replay and fail-closed handling for workers that cannot cooperate. The existing terminal cancel/terminate operations retain their contracts. Capability publication requires exact published Server and SDK conformance.

[rmcdaniel]

Next implementation scope: durable delivery before the service request API

The completed deadline patch bounds portable lease/command authority, but exposing WorkflowStub::attemptRequestCancellation() alone would still leave SDK delivery and replay incorrect. The engine records CooperativeCancellationRequested at acceptance and CooperativeCancellationDelivered at a particular durable call sequence. That distinction must survive portable execution.

Shared contract

Boundary	Required behavior
Request	A separate operation accepts one original request and immutable cleanup deadline. A duplicate returns that same identity/deadline. Acceptance leaves the run active. Terminal cancel/terminate retain their existing semantics.
Observation	Claimed-task payload and successful heartbeat expose the same pending request identity, acceptance time and deadline. A request arriving after claim becomes visible without surrendering valid cleanup authority. Existing terminal HTTP409 stop responses remain distinct.
Delivery	An engine-validated portable delivery marker binds the original request to the eligible authored durable-call sequence and call kind. Persist canonical CooperativeCancellationDelivered once. A transport boolean must not cause replay to throw earlier than that recorded boundary.
Replay	Already recorded results preceding the request replay normally. Cold PHP/Python/Rust workers deliver at the same recorded boundary, including after a worker dies between observation, delivery acknowledgment and cleanup submission.
Cleanup	Preserve the embedded bounded cleanup/shielding semantics. Delivery cancels the affected open wait; permitted cleanup can submit durable work before the deadline. Cleanup completion records cancellation with the original request rather than ordinary workflow success. Deadline expiry and termination revoke remaining authority.
Worker compatibility	Reserve an additive protocol/capability evolution for cooperation. Requested runs cannot be leased to workers that lack it. Refuse a request against an incompatible active workflow-task claim without partially recording it. Capability proof must bind to the claim/registration identity, so later re-registration cannot make an old executing worker appear capable.

The next engine change is portable delivery-marker validation and canonical history, followed by Server request/observation/routing and each SDK's replay handling. Keep the service capability unsupported until the exact published Server and all three SDK artifacts qualify.

Qualification cases

For each workflow language, cover request before claim, request during an active claim/local activity, an already waiting run, duplicate requests, prior completed calls, delivery acknowledgment loss, cold restart during shielded cleanup, cleanup deadline expiry and termination during cleanup. Check original run/request IDs, identical delivery sequence, exactly one terminal history outcome, revoked late local/remote completion and no duplicate committed cleanup work. Include mixed old/new workers, re-registration and a request/claim race. External effects remain at least once and need idempotency or reconciliation.

Reuse the engine deadline corpus and the lifecycle runner, adding actual per-language execution and scenario outcomes. Successful package resolution or an older terminal-cancel pass does not fill this cooperative contract.

[rmcdaniel]

Workflow PR597 now implements a reproduced prerequisite: a cancellation delivery before an activity/wait is scheduled must still consume that authored call position. Main assigned the following portable cleanup command the same sequence. The regression failed before the fix and passes afterward. A database-backed bridge case confirms cleanup is assigned the next sequence, and 69 focused sequence/cold-replay/local-activity tests pass. The distinct service request API remains pending here. Next on this PR is the retained regression corpus and required CI, then the portable delivery-marker implementation.

[rmcdaniel]

Delivered Workflow 2.2.23 at source 9a1400ea8a33fcf7599ef199737d2861a7020c44.

Main CI passed source quality, replay and counterfactual corpus, every MySQL/Postgres/MariaDB feature shard, supported Laravel source upgrades and coverage. Published verification passed package/mirror checks and all sixteen Laravel 9–13 / PHP 8.1–8.4 upgrade cells.

A fresh Composer project installed exactly 2.2.23 from Packagist at that reference. Reflection verified execution from the installed package. The delivery cursor reserved the interrupted call position and cold replay emitted the expected cleanup activity. Retained result, SHA256 e25865e55bca5a1640b78b19a4124a9a4b3b92b0be3af110ed079c18324304c3.

The merged PR branch is absent. Current consumer decisions remain as recorded: stable Server cancellation does not enter this cooperative path, Waterline observes history rather than submitting cleanup, and existing Sample App/Cloud service workflows do not expose the additive request. The upcoming portable delivery implementation must require this corrected engine.

This delivers the sequence prerequisite. Organization issue136 remains open. Next action is the engine-validated delivery marker, then additive Server request/observation/routing, SDK replay and exact published cross-language conformance. Existing terminal cancellation semantics and capability claims are unchanged.

[rmcdaniel]

The portable engine delivery implementation is now in draft
durable-workflow/workflow#598, head eb47ce0d.

It validates the original request and leased task, persists one delivery marker
at an eligible authored call, fences the interrupted activity/timer and preserves
the workflow lease for cleanup. It reserves the full parallel range and supports
selection handles, including nested members and already-resolved operations.
Cleanup return, failure or attempted continuation keeps the original cancelled
outcome. Existing custom bridges retain their original interface.

Local checks pass: 32 persisted database cases / 196 assertions, 98 focused
unit/replay/protocol cases / 1,632 assertions, 64 corpus-policy tests, ECS and
PHPStan. The official baseline/current corpus comparison passes with one new
counterfactual fixture. Normal MySQL/PostgreSQL, coverage and source-upgrade
checks are running on the PR. Protocol 1.20 names the additive history fields.

Next: finish those engine checks and publish the minor integration release, then
implement Server request/observation and claim-bound capability routing. The
PHP/Python/Rust replay, registration races and exact published-tuple lifecycle
qualification remain part of this issue. No service capability is advertised
from the engine bridge alone.

[rmcdaniel]

Protocol review correction: head ba580a6e keeps the existing worker wire
protocol at 1.19. This PR's engine bridge is an in-process integration API and
does not add the service request/delivery endpoints or advertise a cooperative
worker capability. The immutable protocol catalog remains unchanged. The
Server/SDK follow-up owns the new wire version and its versioned spec carriers.

The source release metadata selects Workflow 2.3.0 for the additive bridge API.
The expanded local specification/unit/replay checks pass: 114 tests / 2,351
assertions. The earlier PR head passed MySQL, quality and corpus CI, and exposed
the premature wire-version bump. The corrected head is being checked with both
the PR gates and the full supported-backend/coverage/source-upgrade workflow.
The complete service lifecycle and exact published PHP/Python/Rust tuple remain
unfinished on this issue.