GHCR `eval-agent-server` has 133k+ tags with no retention; tag scheme fans out per SDK sha

[VascoSch92]

Problem

The package ghcr.io/openhands/eval-agent-server currently holds 133,445 tags with no retention policy, accumulated over ~5 months. The tag scheme {sdk_sha}-…-{bench}-{repo}-source-minimal fans out on every SDK commit, so each new sha triggers a full rebuild+push of the whole benchmark dataset. This is both an ergonomic problem (GHCR UI and tag APIs are unusable at this scale) and a concrete pipeline problem (builds now hit ENOSPC on GH-hosted runners).

The package is public, so this is not a billing/quota issue — but it is actively hurting the build pipeline.

Evidence (as of 2026-04-21)

Metric	Value
Total tags	133,445
SDK-sha image tags	130,602 (97.9%)
buildcache-* tags	2,793
Legacy v1.0.0* tags	30
Distinct SDK shas producing tags	548
SDK shas with ≥ 1,000 tags each	63
Oldest SDK sha still present	db900ea — committed 2025-11-24
Median per-tag compressed size	1.38 GB (multi-arch index)
Buildcache median size	2.03 GB

Benchmark distribution (substring match):

• swebench: 130,304 tags (97.6%)
• commit0: 1,435
• gaia: 192

Downstream effect

This directly causes the build hangs we've been seeing. Example: eval run OpenHands/evaluation Actions run 24706920746 (commit0 × Gemini 3.1 Pro) has been stuck in Build Commit0 Images / build-and-push for 4h45m+ because its SDK sha aabf4072… has no cached tags in the registry → BuildKit has to build + push the full image set → GH ubuntu-24.04 runner (~14 GB free) hits ENOSPC → BuildKit hangs instead of failing fast.

The prior failed run on the same branch (24682483119) has this in its manifest.jsonl artifact:

"built", "docker.io/wentingzhao/minitorch:v0", "error": "OSError(28, 'No space left on device')"

Reproduce the numbers

Anonymous (no GH token needed; package is public):

TOKEN=$(curl -sS 'https://ghcr.io/token?service=ghcr.io&scope=repository:openhands/eval-agent-server:pull' | jq -r .token)

# total tag count (paginate via Link header; n=1000 per page)
curl -sS -H "Authorization: Bearer $TOKEN" \
  'https://ghcr.io/v2/openhands/eval-agent-server/tags/list?n=1000'

# inspect a manifest / size
curl -sS -H "Authorization: Bearer $TOKEN" \
  -H 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.docker.distribution.manifest.v2+json' \
  'https://ghcr.io/v2/openhands/eval-agent-server/manifests/<TAG>'

UI: https://github.com/orgs/OpenHands/packages/container/package/eval-agent-server

Proposed fix

Two changes, in order of impact:

1. Retention / GC workflow (execution fix). Nightly or weekly job that deletes versions where:

   • the SDK sha is NOT one of the last N (say 30) shas on software-agent-sdk main, AND
   • no metadata entry in gs://openhands-evaluation-results/metadata/*.jsonl with a non-terminal status references that sha.

   Dry-run first. On first real pass this will likely purge the majority of the 130k tags. Buildcache tags can be pruned more aggressively (keep last 2–3 per {bench}/{repo}).

2. Split the image (structural fix). The tag scheme invalidates everything on every SDK commit. Bake a stable per-repo base ({ext_sha}-{bench}-{repo}) and produce a thin SDK-only layer on top. New SDK commits then push MBs, not GBs, which also eliminates the ENOSPC hang on GH runners as a side effect.

Cross-ref: this investigation touched OpenHands/evaluation side too (register_metadata.py, kill-eval-job.yml), so any retention workflow will want to coordinate with the metadata jsonl schema defined there.

cc @juanmichelini @simonrosenberg

[juanmichelini]

@simonrosenberg @VascoSch92 is any of you able to:

Go to: https://github.com/orgs/OpenHands/packages/container/eval-agent-server/settings
Look for "Retention" settings
Set a retention period (e.g., 14 or 30 days)

?

[VascoSch92]

@juanmichelini I don't have permissions for that

[juanmichelini]

@simonrosenberg do you have permissions?

[simonrosenberg]

@simonrosenberg do you have permissions?

No!

[VascoSch92]

Status update + next-step playbook

Current state

Retention workflow (.github/workflows/ghcr-retention.yml) is in flight at #689. All three pieces of plumbing work:

• PAT scoping (read:packages + delete:packages) — verified
• Workflow auth, filters, selection logic — verified via dry-run (selects 378 tagged + 2692 untagged for deletion while preserving 100 most-recent and v1.0.0*)
• Action cannot finish the job on a package this large, for reasons documented below

Empirical tag count on 2026-04-23 via the Docker Registry v2 API: 135,133 tags (up from ~133k in the issue body — CI is still pushing).

What blocks us

Two reproducible failures on the current (large) package:

1. Run 24822044827 — GitHub Packages API returned {"message":"Server Error"} (HTTP 500) during pagination after ~89 min. snok couldn't deserialize and crashed.
2. Run 24825732828 — pagination succeeded this time, but snok panicked at src/core/delete_package_versions.rs:41 with OutOfRangeError(()). Root cause: the action captures rate_limit_reset at the start of enumeration, then later computes rate_limit_reset - Utc::now() and unwraps .to_std(). Our enumeration took 97 min — longer than the 1h rate-limit window — so the subtraction yielded a negative chrono::Duration, which .to_std() refuses to convert to an unsigned std::time::Duration. Filed upstream: Panic at delete_package_versions.rs:41 when enumeration exceeds rate-limit window (to_std().unwrap() on negative Duration) snok/container-retention-policy#128.

The panic fires before the tagged-version deletion pass, so even a dry-run: false run on the current state would at best delete some untagged orphan layers, never touch the tagged backlog, and crash with no final log.

Next step: GitHub Support bulk-delete

Public tooling cannot drain 135k versions in a single rate-limit window and the upstream action bug blocks iteration. Escape hatch is asking GitHub Support to run an internal bulk-delete.

Action item open a ticket at https://support.github.com/contact → Category: GitHub Packages → Container registry (ghcr.io) → Severity: Normal.

Suggested body:

Our eval-agent-server container package in the OpenHands org has ~135,133 versions (empirical count via Registry v2 API, 2026-04-23) from per-commit tag fan-out in CI. We cannot drain this via the public REST API:

• 5,000 req/hr rate limit can't cover enumeration + deletion in one window
• snok/container-retention-policy panics with OutOfRangeError on packages large enough that enumeration exceeds the rate-limit reset — filed at Panic at delete_package_versions.rs:41 when enumeration exceeds rate-limit window (to_std().unwrap() on negative Duration) snok/container-retention-policy#128
• Direct pagination hits intermittent Server Error 500s (reproduced in https://github.com/OpenHands/benchmarks/actions/runs/24822044827)

Requesting an internal bulk-delete on ghcr.io/openhands/eval-agent-server under this policy:

• Delete versions with updated_at older than 2 months (before 2026-02-23)
• Preserve any version tagged v1.0.0*
• Preserve the 100 most-recently-updated versions (safety buffer)
• Include both tagged and untagged versions

Once the package is back to a manageable size, our nightly retention workflow (.github/workflows/ghcr-retention.yml) will keep it healthy going forward.

Expected turnaround: 1–3 business days. First reply is usually a confirmation request — this is destructive and irreversible, so they'll want sign-off in writing before executing.

After Support drains the backlog

Nightly workflow takes over. It has already demonstrated correct selection logic on a dry-run. Remaining cleanups once the backlog is gone:

• Upstream fix lands for snok#128 (or we pin to a bugfix SHA / switch actions). Tracked there; not blocking us once the package is small.
• Optional: cross-reference eval metadata on GCS (gs://openhands-evaluation-results/metadata/*.jsonl) before deleting, so an in-flight eval against a >2mo SDK sha can't lose its image mid-run. Low probability given the 100-version buffer, but worth a follow-up PR.

[VascoSch92]

It seems that a part of the issue was fixed by the PR: #690