Repository navigation
fix(ci): retry a throttled pull in the gosu-skip check - #1253
Merged
Merged
Conversation
ghcr.io answers toomanyrequests to shared CI runners, and check_gosu_skip.sh pulled once and failed. That turned "Validate configs" red on main for 4eeeffa and ce42629 (2026-10-21/22) and on #1243, and a red tip makes converge.sh refuse to deploy, so both hosts' converge timers stopped and ScheduledJobStale fired on trinity for a commit that was fine. pull_with_retry retries only on toomanyrequests, after 5, 20 and 60 s — check_compose_health.py's PULL_RETRY_PAUSES (#602) — and fails at once on anything else. Tested with a fake docker: throttled twice then served passes on the third try, a manifest error fails on the first, and a throttle that never lifts gives up after four. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3 tasks
There was a problem hiding this comment.
🟢 Approval recommended
The narrowly scoped retry logic preserves failure handling, and isolated verification confirmed its retry bounds and success and failure paths.
No open issues 🎉
What changed in this PR
Adds bounded retries to the gosu-skip image check to reduce CI failures caused by registry throttling, which can block deployment convergence.
Changes:
- Retries
toomanyrequestspull failures after 5, 20, and 60 seconds. - Preserves immediate failure for other pull errors and reports the final error when retries are exhausted.
| File | Description |
|---|---|
scripts/check_gosu_skip.sh |
Adds a throttling-aware pull helper and uses it in the image-check loop. |
Review effort: Balanced
check_image_pins.py traces a docker command's image by variable name back to image-for.sh. pull_with_retry pulled "$1", which it cannot trace, and CI's "Verify every docker image comes from compose.yaml" failed on it. The argument is now `local img`, the name the loop already resolves from compose.yaml. check_image_pins.py: 71 invocations OK; the retry's fake-docker cases still pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
scripts/check_gosu_skip.shnow retries an image pull when the registry answerstoomanyrequests, after 5, 20 and 60 seconds (the pausescheck_compose_health.pyalready uses, #602). Any other pull failure still fails at once.Why
ghcr.io throttles shared CI runners, and this check pulled once. The resulting red "Validate configs" hit
mainat 4eeeffa and ce42629, and #1243. A red tip makesconverge.shrefuse to deploy, so both hosts' converge timers stopped andScheduledJobStalefired on trinity for commits that were fine.Blast radius
CI and
make validateonly. At most 85 s more on a throttled run; no change when the registry answers.secrets/*.sops.yamlVerification
./scripts/check_gosu_skip.shon trinity: OK, 3 pinned images.pull_with_retryagainst a fakedocker:manifest unknown: fails on try 1;shellcheck: clean.
make validatepasses — the touched check was run on its own, plus shellcheckDeployed to the lab and confirmed working — CI-only change
Docs updated (
docs/, service README, ordocs/roadmap.md) — nothing to update🤖 Generated with Claude Code