Worker pool silently dies under load (500ms response-timeout → fatal StreamError) and on restart (register_worker self-re-registration guard)

[glani]

Summary

Running a Monitor with N RedisStorage-backed workers, the entire worker pool stops on its own under two independent conditions, both rooted in apalis-redis. After the workers stop, Monitor::run_all_workers calls start_shutdown(), so the process keeps running with a silently dead job pipeline — no panic, no shutdown signal, every worker just emits Event::Stop.

Versions: apalis-core 1.0.0-rc.9, apalis-redis 1.0.0-rc.8, redis 1.2.x.

Trigger 1 — 500 ms default response timeout becomes a fatal StreamError under load

apalis_redis::connect() builds the manager via Client::get_connection_manager(), which uses redis-rs's DEFAULT_RESPONSE_TIMEOUT = Some(500ms). A pool multiplexes roughly workers × concurrency commands over the one ConnectionManager; under load a single redis stall > 500 ms makes the in-flight poll/fetch return io::ErrorKind::TimedOut. apalis-core maps that to WorkerError::StreamError, which is fatal — run_all_workers has no retry for a stream error, so the worker future returns Err. Because several in-flight commands stall together, the whole pool dies at once:

ExitError(MonitoredErrors:
 - Worker `q0`: Failed to consume task stream: timed out
 - Worker `q1`: Failed to consume task stream: timed out
 ...
)

Reproduce deterministically by constructing the manager with a small response_timeout (e.g. 50 ms) and pushing a steady load — the pool dies within seconds-to-minutes.

Two issues here: (a) connect() gives no way to set/disable the response timeout, and a 500 ms per-command default is surprising for a job backend; (b) a transient poll error terminates the worker instead of being retried.

Trigger 2 — register_worker.lua rejects self-re-registration → restart/​respawn dies instantly

lua/register_worker.lua raises error("worker is still active within threshold") when now - last_seen < threshold, with threshold == keep_alive. The steady-state heartbeat can't trip it (its sleep is never early, so now - last_seen ≥ threshold), but a process restart or a Monitor restart re-registers the same stable worker names while the previous entries are still fresh (< keep_alive old). The Lua error() surfaces as a fatal worker-stream failure, so the restarted pool dies on its first poll.

Reproduce deterministically: start the pool, kill it, restart within keep_alive → all workers Event::Stop ~1 ms after start.

A worker refreshing its own liveness should never be rejected — it's a keep-alive, not a duplicate claim.

Suggested fixes

1. Let connect() / RedisStorage configure the ConnectionManager response_timeout (and consider defaulting it off for the backend); and/or have apalis-core retry the poll on a transient backend error rather than terminating the worker.
2. In register_worker.lua, allow a worker to refresh its own last_seen (only reject a genuinely foreign claim, if at all).

Workaround

Vendored apalis-redis with connect() setting response_timeout = None and register_worker.lua dropping the self-re-registration guard; plus a supervisor that respawns the Monitor on an unexpected stop. Soak: > 30 min under load with zero deaths.

Happy to send a PR for either fix.

[geofmureithi]

I need to look a little into this, sounds like something is not right with the method run_all_workers.

Regarding this:

In register_worker.lua, allow a worker to refresh its own last_seen (only reject a genuinely foreign claim, if at all).

This is done to prevent registering the worker twice (within the heartbeat period) because we cannot verify that the worker is not dead. This is also used to find orphaned jobs (jobs where their worker crashed before completion) and return them to the queue.

Regarding connection:

connect()

connect() is supposed to be a helper with defaults, but you can create you own Connection object from the redis-rs crate.
RedisStorage allows you to pass in a custom connection type:
https://docs.rs/apalis-redis/1.0.0-rc.8/apalis_redis/struct.RedisStorage.html#method.new

[github-actions]

This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs within 7 days.
Thank you for your contributions.

[Mobinkh]

hi im facing same problem randomly in docker containers i think its exactly related to docker clock shifts and heartbeat timeout. im so hyped to see a fix for this.

[geofmureithi]

This should be resolved with #102