test(gasoline): reproduce slow workflow pulls on postgres

[MasterPtato]

No description provided.

[MasterPtato]

Stack for rivet-dev/actors

• #5717 ci(publish): allow release-mode builds for preview dispatche …
• #5716 fix(pegboard-envoy): use async scc methods so actor stop can …
• #5704 fix(universaldb): run postgres transaction reads concurrentl …
• #5705 test(gasoline): reproduce slow workflow pulls on postgres 👈
• #5702 fix(universaldb): chunk postgres commit requests that exceed …
• #5703 fix(universaldb): clear postgres key ranges one statement pe …
• #5706 perf(universaldb): time each drain batch statement to locali …
• #5695 fix(universaldb): report the last error when transaction ret …
• main

Get stack: forklift get 5705
Push local edits: forklift submit
Merge when ready: forklift merge 5705

_{change qnlwwlpm}

[the-company-company]

✅ No issues found

_{Reviewed commit 39084c5.}

[claude]

Review

This PR adds a single #[ignore]d investigation harness (engine/packages/gasoline/tests/pull_backlog.rs) that reproduces slow pull_workflows calls under Postgres with added network latency. It's diagnostic-only (no production code changes), well documented, and correctly placed under tests/ per the repo's Rust test-layout convention.

I traced every API call against the current gasoline source to check for drift:

• dispatch_workflow(ray_id, workflow_id, workflow_name, tags, input, unique), update_worker_ping(worker_id, worker_version, update_active_idx), and pull_workflows(worker_id, worker_version, filter, running_workflows_by_name) all match the Database trait signatures in db/mod.rs.
• The PHASE_SPANS names (pull_workflows, read_wake_conditions, map_to_leased_workflows, pull_workflows_tx, clear_workflow_secondary_idx_tx, pull_workflow_history_tx) all correspond to real #[tracing::instrument]/debug_span!/info_span! names in db/kv/mod.rs.
• All crates used (rivet-test-deps, futures-util, serde_json, tracing-subscriber, uuid, etc.) are already in gasoline's [dependencies], so no Cargo.toml changes are needed.
• Formatting uses hard tabs throughout, matches rustfmt.toml.
• The doc comment's claim ("a worker gives up when pull_workflows takes longer than its fixed pull timeout, and giving up stops every workflow it is running") matches worker.rs's tick(), which wraps pull_workflows in tokio::time::timeout(PULL_WORKFLOWS_TIMEOUT, ...) and propagates the error up through shutdown().
• The docker run --net container:<name> --cap-add NET_ADMIN alpine:3 ... sidecar passes container_name as a discrete arg (not interpolated into a shell string), and the only value spliced into the inner sh -c string (rtt_ms) is a parsed u64, so there's no command-injection concern here despite the string formatting.

Minor / optional suggestions

1. Duplicated timeout constant risk of silent drift. PULL_WORKFLOWS_TIMEOUT is redefined here as Duration::from_secs(10) because the real one (worker.rs:30) is pub(crate) and not reachable from an integration test. The doc comment already explains this, so it's not hidden, but if the worker's real timeout ever changes, this harness's pass/fail threshold goes stale silently. Worth a tracking note, or exposing the constant behind a test-utils accessor if this harness is expected to be maintained long-term.

2. Hardcoded container suffix. add_network_latency(&format!("test-postgres-{test_id}-1"), rtt_ms) hardcodes the -1 datacenter-label suffix. It happens to be correct today because TestDeps::new_with_test_id always sets up a single datacenter with label 1 (per test-deps/src/lib.rs), but spelling it as config.dc_label() instead of a literal would make that assumption self-documenting and resilient if the harness is ever adapted to a multi-DC setup.

3. No assertion on pulled count, only on elapsed time. That's presumably intentional for an investigation tool (partial pulls due to the concurrency quota are expected), but a one-line comment on why pulled.len() isn't checked against backlog would save a future reader from wondering.

None of these are blocking; the harness is self-contained, #[ignore]d (won't affect CI), and doesn't touch any load-bearing code.

🤖 Generated with Claude Code