feat(job): run Backup Site on a dedicated queue and worker (backport #567) - #569
Merged
Conversation
Backup Site shared the `low` queue with New Bench, Archive Bench, Update Site Pull/Migrate and binlog uploads, and every worker listened on `high default low`. A long backup could occupy all workers and stall everything else behind it. Move Backup Site to its own `backup` queue with a dedicated `worker_backup` supervisor program. The main workers are unchanged and never pick up backup jobs. The backup worker lists `backup high default low` so it drains backups first and only helps with the shared queues while idle. Worker count is configurable via `backup_workers` in the agent config, defaulting to 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit ff664dd)
update_agent_web ran a bare `supervisorctl reread`, which loads the new config but never adds or starts programs that are new in the template. Existing installations would regenerate supervisor.conf with worker_backup in it and then never run it, so backup jobs would sit in the queue forever. Use the existing _update_supervisor() helper (reread + update) that setup_supervisor and update_agent_cli already rely on. Also restart/stop/start the backup workers alongside the main ones in update_agent_web and update_agent_cli, otherwise they keep running stale code across agent updates. Tolerate failure there so an agent updating from a config without worker_backup doesn't break. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> (cherry picked from commit bae26f7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Backup Sitewas enqueued withpriority="low", sharing thelowqueue withNew Bench,Archive Bench,Update Site Pull,Update Site Migrateand the binlog upload jobs. Every worker insupervisor.conflistens onhigh default low, so a few concurrent long backups can occupy all{{ workers }}processes and stall everything else behind them.Change
Backup Sitemoves to its ownbackupqueue (agent/site.py).[program:worker_backup]insupervisor.conf.jinja2, count configurable viabackup_workersin the agent config (default1).[program:worker]is untouched and never picks up backup jobs.worker_backuplistens onbackup high default low— RQ pops in the listed order, so backups always win, and the worker helps with the shared queues rather than idling between backups.Procfilemirrors this for local dev.Physical Backup Databasedeliberately stays onlow.No allowlist / request-param plumbing is needed here: the queue name is static in the
@jobdecorator, not caller-supplied, soqueue("backup")inagent/job.pyworks as-is. This is complementary to #557, which routes the same job type to different queues per call — if that lands first,"backup"just joins itsQUEUEStuple.Known ceiling
A long
lowjob picked up byworker_backupwill delay a backup enqueued behind it. Noted in a comment in the template; drop the trailing queues from that worker's queue list if backup latency ever matters more than keeping the worker busy.Rollout
Deploy so the supervisor config is regenerated before or with the decorator change — jobs enqueued to
backupsit unprocessed until the new worker exists.Verification
Rendered the template and asserted the queue wiring:
```
main worker -> rq worker ... high default low (unchanged, no backup)
backup worker-> rq worker ... backup high default low
programs=web, redis, worker, worker_backup
```
Note: pre-commit's ruff hook fails on
agent/site.pyondevelopalready (B904 at line ~1318, plus format drift, same pinned ruff 0.13.1) — pre-existing from #553, untouched here. This commit skips hooks to avoid dragging unrelated reformatting into the diff.🤖 Generated with Claude Code
This is an automatic backport of pull request #567 done by Mergify.