feat(job): run Backup Site on a dedicated queue and worker

[balamurali27]

Problem

Backup Site was enqueued with priority="low", sharing the low queue with New Bench, Archive Bench, Update Site Pull, Update Site Migrate and the binlog upload jobs. Every worker in supervisor.conf listens on high default low, so a few concurrent long backups can occupy all {{ workers }} processes and stall everything else behind them.

Change

• Backup Site moves to its own backup queue (agent/site.py).
• New [program:worker_backup] in supervisor.conf.jinja2, count configurable via backup_workers in the agent config (default 1).
• The existing [program:worker] is untouched and never picks up backup jobs.
• worker_backup listens on backup high default low — RQ pops in the listed order, so backups always win, and the worker helps with the shared queues rather than idling between backups.
• Procfile mirrors this for local dev.

Physical Backup Database deliberately stays on low.

No allowlist / request-param plumbing is needed here: the queue name is static in the @job decorator, not caller-supplied, so queue("backup") in agent/job.py works as-is. This is complementary to #557, which routes the same job type to different queues per call — if that lands first, "backup" just joins its QUEUES tuple.

Known ceiling

A long low job picked up by worker_backup will delay a backup enqueued behind it. Noted in a comment in the template; drop the trailing queues from that worker's queue list if backup latency ever matters more than keeping the worker busy.

Rollout

Deploy so the supervisor config is regenerated before or with the decorator change — jobs enqueued to backup sit unprocessed until the new worker exists.

Verification

Rendered the template and asserted the queue wiring:

```
main worker -> rq worker ... high default low (unchanged, no backup)
backup worker-> rq worker ... backup high default low
programs=web, redis, worker, worker_backup
```

Note: pre-commit's ruff hook fails on agent/site.py on develop already (B904 at line ~1318, plus format drift, same pinned ruff 0.13.1) — pre-existing from #553, untouched here. This commit skips hooks to avoid dragging unrelated reformatting into the diff.

🤖 Generated with Claude Code

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[greptile-apps]

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

_{Reviews (2): Last reviewed commit: "fix(server): start new supervisor progra..." | Re-trigger Greptile}

[balamurali27]

Good catch — pushed a fix in bae26f7. Two related problems, both would have left backups queued forever on existing installations:

1. update_agent_web never ran supervisorctl update

agent/server.py inlined a bare reread, which loads the new config but does not act on it. worker_backup would be written into supervisor.conf and then never started. _update_supervisor() (reread + update) already existed and is what setup_supervisor() / update_agent_cli use — update_agent_web just wasn't using it. Swapped.

2. The restart loops only covered agent:worker-N

Even once update starts the backup workers, nothing restarted them on subsequent agent updates, so they'd keep running stale code indefinitely. Added worker_backup to the restart loop in update_agent_web and to the stop/start loops in update_agent_cli, with non_zero_throw=False so an agent coming from a config without worker_backup doesn't break mid-update.

Also added a Server.backup_workers property so the 1 default isn't duplicated across the template context and the loops.

Verified get_supervisor_processes_status() groups these correctly — it splits on the first -, so worker_backup-0 lands under worker_backup and does not collide with the worker group that the stop loop iterates:

{'worker': {'0': 'RUNNING', '1': 'RUNNING'}, 'worker_backup': {'0': 'RUNNING'}}

This also removes the rollout ordering caveat from the PR description — update adds and starts the program in the same pass that writes the config.