Skip to content

fix(envoy-client): signal lost actor generations synchronously - #5821

Open
abcxff wants to merge 1 commit into
stack/fix-rivetkit-isolate-workflow-run-state-per-actor-generation-wurkyvuqfrom
stack/fix-envoy-client-signal-lost-actor-generations-synchronously-ykpvpvyo
Open

abcxff wants to merge 1 commit into
stack/fix-rivetkit-isolate-workflow-run-state-per-actor-generation-wurkyvuqfrom
stack/fix-envoy-client-signal-lost-actor-generations-synchronously-ykpvpvyo

Conversation

@abcxff

@abcxff abcxff commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

No description provided.

@abcxff

abcxff commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

@claude

claude Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Review: fix(envoy-client): signal lost actor generations synchronously

The design is sound. A per-generation CancellationToken is cancelled synchronously on both loss paths (engine Stop(Lost) and the envoy lost-threshold timer). Late KV writes are fenced at the transmit point. Stopped(Ok) is prevented from escalating into a destroy. The tests cover the main races.

Potential issues

  1. Token not cancelled on every Stop(Lost) path. In actor.rs, the Stop arm only calls ctx.lost.cancel() when a stop is already pending. Otherwise it relies on commands.rs having cancelled the token first. A ToActor::Stop { reason: Lost } sent any other way leaves the token uncancelled until begin_stop. Cancelling in the actor task whenever reason == Lost would be self-consistent and is idempotent.
  2. kv_put_fenced has no callers in this PR. on_actor_start_with_lost_signal and kv_put_fenced are not consumed anywhere in the diff, so the fence has no effect yet. Please confirm a follow-up exists. Only put is fenced; delete and other mutating KV ops are not.
  3. The fence covers only unsent requests. A request already on the wire can still be applied. The doc comments should say so.
  4. Behavior change. The local lost-threshold stop now passes StopActorReason::Lost instead of SleepIntent to on_actor_stop. Worth noting in the PR description.
  5. API duplication. insert_actor wraps insert_actor_with_lost_signal with a fresh unconnected token, which silently disables the fence. Consider whether non-test callers should all use the lost-signal variant.

Style / conventions

  • Missing blank line before #[tokio::test] async fn lost_signal_cancels_the_generation_token.
  • New actor tests are added to the inline mod tests in src/actor.rs. CLAUDE.md asks for Rust tests under tests/. The module is pre-existing, so this is acceptable if private access is needed.
  • The PR description is empty. CLAUDE.md asks for a short bullet list.
  • Per the runner parity rule, check whether the TypeScript envoy client needs the same behavior, or document the gap.

Test coverage

Good coverage of the new logic. Missing cases:

  • A kv_put_fenced end-to-end test through EnvoyHandle.
  • A test for the lost-threshold timer cancelling entry.lost.
  • Resend after reconnect for a lost-cancelled request.

No security or performance concerns. The fence check is a single atomic load on the send path.

@the-company-company the-company-company Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 1 high-severity finding

Reviewed commit bb0f509.

Comment on lines 80 to +84
if request.sent {
return;
}
// A request queued before its generation was declared lost must not reach storage, even
// when it is resent after a reconnect.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 High · Make the lost check atomic with WebSocket admission

This check runs before ws_send awaits shared.ws_tx.lock(), so the generation can be marked lost while the send is blocked and the stale KvPut is still admitted after the lock is released. That defeats the fence exactly when an old generation is being revoked and can let it write concurrently with its replacement. Pass the token into a cancellation-aware send path and recheck it after acquiring the WebSocket lock, immediately before queueing the frame; add a regression test that holds the lock, starts the fenced send, cancels the token, then releases the lock and asserts that no frame is emitted.

@abcxff
abcxff force-pushed the stack/fix-rivetkit-isolate-workflow-run-state-per-actor-generation-wurkyvuq branch from d389d55 to 021b0ee Compare October 3, 2026 02:05
@abcxff
abcxff force-pushed the stack/fix-envoy-client-signal-lost-actor-generations-synchronously-ykpvpvyo branch from bb0f509 to 14f8833 Compare October 3, 2026 02:05
@abcxff
abcxff force-pushed the stack/fix-rivetkit-isolate-workflow-run-state-per-actor-generation-wurkyvuq branch from 021b0ee to 399ced5 Compare October 6, 2026 01:29
@abcxff
abcxff force-pushed the stack/fix-envoy-client-signal-lost-actor-generations-synchronously-ykpvpvyo branch from 14f8833 to 5a2d258 Compare October 6, 2026 01:29

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant