Skip to content

fix(ci): retry a throttled pull in the gosu-skip check - #1253

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/gosu-pull-retry
Oct 22, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/gosu-pull-retry

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Oct 22, 2026

Copy link
Copy Markdown
Owner

What changed

scripts/check_gosu_skip.sh now retries an image pull when the registry answers toomanyrequests, after 5, 20 and 60 seconds (the pauses check_compose_health.py already uses, #602). Any other pull failure still fails at once.

Why

ghcr.io throttles shared CI runners, and this check pulled once. The resulting red "Validate configs" hit main at 4eeeffa and ce42629, and #1243. A red tip makes converge.sh refuse to deploy, so both hosts' converge timers stopped and ScheduledJobStale fired on trinity for commits that were fine.

Blast radius

CI and make validate only. At most 85 s more on a throttled run; no change when the registry answers.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

  • ./scripts/check_gosu_skip.sh on trinity: OK, 3 pinned images.

  • pull_with_retry against a fake docker:

    • throttled twice, then served: passes on try 3;
    • manifest unknown: fails on try 1;
    • always throttled: gives up after 4 tries.
  • shellcheck: clean.

  • make validate passes — the touched check was run on its own, plus shellcheck

  • Deployed to the lab and confirmed working — CI-only change

  • Docs updated (docs/, service README, or docs/roadmap.md) — nothing to update

🤖 Generated with Claude Code

ghcr.io answers toomanyrequests to shared CI runners, and check_gosu_skip.sh
pulled once and failed. That turned "Validate configs" red on main for
4eeeffa and ce42629 (2026-10-21/22) and on #1243, and a red tip makes
converge.sh refuse to deploy, so both hosts' converge timers stopped and
ScheduledJobStale fired on trinity for a commit that was fine.

pull_with_retry retries only on toomanyrequests, after 5, 20 and 60 s —
check_compose_health.py's PULL_RETRY_PAUSES (#602) — and fails at once on
anything else. Tested with a fake docker: throttled twice then served
passes on the third try, a manifest error fails on the first, and a
throttle that never lifts gives up after four.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 22, 2026 17:53

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The narrowly scoped retry logic preserves failure handling, and isolated verification confirmed its retry bounds and success and failure paths.

No open issues 🎉

What changed in this PR

Adds bounded retries to the gosu-skip image check to reduce CI failures caused by registry throttling, which can block deployment convergence.

Changes:

  • Retries toomanyrequests pull failures after 5, 20, and 60 seconds.
  • Preserves immediate failure for other pull errors and reports the final error when retries are exhausted.
File Description
scripts/​check_gosu_skip.sh Adds a throttling-aware pull helper and uses it in the image-check loop.

Review effort: Balanced

check_image_pins.py traces a docker command's image by variable name back to
image-for.sh. pull_with_retry pulled "$1", which it cannot trace, and CI's
"Verify every docker image comes from compose.yaml" failed on it. The
argument is now `local img`, the name the loop already resolves from
compose.yaml. check_image_pins.py: 71 invocations OK; the retry's fake-docker
cases still pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit f846a88 into main Oct 22, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/gosu-pull-retry branch October 22, 2026 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants