Support pause/suspend/interrupt/cancel signature actions in hatchet workflows

[yedidyakfir]

Summary

Signatures currently support a rich set of lifecycle actions — pause (SUSPEND / INTERRUPT), suspend, interrupt, cancel, and resume — implemented on Signature / ChainTaskSignature / SwarmTaskSignature in libs/third-magic/thirdmagic/signature/model.py:160-194 and honored at per-task execution via SignatureLifecycle.should_run_task (libs/mageflow/mageflow/lifecycle/signature.py:78-91).

Hatchet workflows dispatched via MageflowWorkflow do not yet honor these signature-level actions. When a workflow is attached to a signature (see #116 / #118), pausing or canceling that signature should propagate to the running workflow — stopping new tasks from dispatching on SUSPENDED, aborting in-flight execution on INTERRUPTED, and cleaning up on CANCELED.

Features / Details

• Respect SignatureStatus.SUSPENDED on MageflowWorkflow runs: do not dispatch subsequent tasks; invoke on_pause_signature with the current input; leave the workflow in a resumable state
• Respect SignatureStatus.INTERRUPTED on in-flight workflow runs: best-effort abort of the running workflow (analogous to the Signature.interrupt TODO at libs/third-magic/thirdmagic/signature/model.py:172-177)
• Respect SignatureStatus.CANCELED on workflow runs: invoke on_cancel_signature, remove signature state, and stop the workflow
• Support resume on a previously SUSPENDED workflow signature — re-dispatching the workflow should pick up from a resumable state (similar to resume_from_key on signatures)
• Propagate pause/cancel through nested workflow signatures (chain / swarm wrapped in a MageflowWorkflow) so pausing the parent signature pauses the nested execution, consistent with ChainTaskSignature.change_status / SwarmTaskSignature.change_status
• Reuse the existing PauseActionTypes enum (SUSPEND / INTERRUPT) and Signature.pause_task entry point rather than introducing a parallel action set

Use Cases

• A caller that dispatched a workflow with a signature wants to pause the workflow mid-run (e.g. rate-limit backoff, manual hold) — they call Signature.pause_from_key and the running MageflowWorkflow stops dispatching further tasks
• A caller cancels a workflow signature — the running workflow winds down, on_cancel_signature runs, and Redis state is cleaned up
• Resume: a suspended workflow is re-dispatched and continues without re-executing already-DONE sub-tasks (relies on existing idempotency guards — 070f2d4, f58d2b3, 893d8ac)
• Hard stop: INTERRUPT forcibly aborts the workflow's async context rather than waiting for the next task boundary

Implementation Considerations

• SignatureLifecycle.should_run_task already implements the per-task version of this check — the workflow-level equivalent needs to run at workflow-start (and potentially between tasks) rather than only at task entry
• MageflowWorkflow._serialize_input (libs/mageflow/mageflow/clients/hatchet/workflow.py:109-122) is where the workflow-level signature reference should be carried so workflow hooks can resolve and inspect the signature's status
• Signature.interrupt is currently a TODO that delegates to suspend (libs/third-magic/thirdmagic/signature/model.py:172-177) — this issue must either implement real interrupt semantics at the hatchet layer or explicitly document the current suspend-equivalent behavior
• Pausing an in-flight hatchet workflow likely requires hatchet-sdk cancellation APIs — investigate what hatchet exposes for stopping a running workflow from an external actor vs. a cooperative check at task boundaries
• Integrates with Update signature status for hatchet workflow lifecycle #118 (workflow signature status updates) — status transitions must stay consistent across ACTIVE → SUSPENDED → ACTIVE (resume) and ACTIVE → CANCELED paths
• Must preserve existing nested-signature propagation: ChainTaskSignature.change_status / SwarmTaskSignature.change_status fan out to children; wrapping those in a MageflowWorkflow must not break the fan-out

Tasks

• Add a workflow-start status check that honors SUSPENDED / CANCELED / INTERRUPTED (workflow-level analogue of should_run_task)
• Wire on_pause_signature / on_cancel_signature invocation into the MageflowWorkflow entry path
• Implement cooperative suspend: stop dispatching further tasks on SUSPENDED mid-run
• Investigate and implement hatchet-level workflow cancellation for INTERRUPT semantics
• Implement resume path for a SUSPENDED workflow signature, leveraging task-level idempotency so completed children are not re-run
• Ensure pause/cancel propagates correctly through MageflowWorkflow-wrapped chain / swarm signatures
• Unit tests: workflow signature in each of SUSPENDED / INTERRUPTED / CANCELED states yields the expected behavior on dispatch
• Integration test: dispatch a MageflowWorkflow, externally pause_from_key the signature, verify no further tasks dispatch
• Integration test: resume a suspended workflow signature and verify it completes without re-running done tasks
• Update documentation to describe workflow-level pause/suspend/interrupt/cancel semantics

Related

• Support with_signature for hatchet workflow requests #116 — workflow-level with_signature support
• Update signature status for hatchet workflow lifecycle #118 — workflow signature status updates (ACTIVE / DONE / FAILED)
• Cancel Tasks (Complete Deletion) #21 — Cancel Tasks (Complete Deletion) — related but task-level, not workflow-level