fix(ci): queue time must not count against the execution timeout - #190
Merged
Conversation
os#367 finished with no report link. The result was fine — the artifact carried a valid report_url — but nobody was listening by then. caller Wait for e2e: 17:11:55 -> 19:25:54 gave up after 2h caller Publish: 19:25:55 downloaded here central verdict job: 19:44:56 -> 19:45:03 artifact uploaded 19 min LATER The run was dispatched at 17:11 but only started executing at 18:27, after 76 minutes queued behind another PR. The suite then took 75 minutes. The 2h budget started ticking at DISPATCH, so most of it was spent queueing and the caller abandoned a healthy run — then downloaded an artifact that did not exist yet and rendered 'unknown'. With one serialized runner and ~75-minute suites, a PR queued behind a single other PR already exceeds 2h. The timeout was measuring the wrong thing. Now two budgets: while QUEUED the busy-vs-offline liveness guard is the protection, not a clock; the 2h execution budget starts when the first job leaves 'queued'. Also retries the result-artifact download (it is uploaded by the central run's last job) and warns loudly if it is still missing, instead of silently discarding a good result.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
os#367finished with no report link. The result was fine — the artifact carried a validreport_url,459-style totals and all — but nobody was listening by the time it existed.Root cause
The run was dispatched at 17:11 but only began executing at 18:27 — 76 minutes queued behind another PR. The suite then ran ~75 minutes.
The 2h budget started ticking at dispatch, so most of it was consumed by queueing. The caller abandoned a perfectly healthy run, then downloaded an artifact that did not exist yet and rendered
unknown.With one serialized runner and ~75-minute suites, a PR queued behind a single other PR already exceeds 2h. The timeout was measuring the wrong thing — it bounded waiting plus running when it should bound running.
The fix — two budgets
queuedThe log now says so explicitly:
Also — stop discarding good results
The result artifact is uploaded by the central run's last job, so a caller arriving early loses it. The download now retries (6 × 20s) and, if still missing, emits a
::warning::with the run URL rather than silently rendering "unknown".That is what turns "the pipeline is broken" into "the report is late", which is a very different conversation with the team.
Replay
os#367: queued 76 min (no longer fatal) + ran 75 min → would have completed and posted its report.🤖 Generated with Claude Code