fix(test): make cd_boot_matrix build stamps converge across chunked runs - #227
Open
JoeMatt wants to merge 1 commit into
Open
fix(test): make cd_boot_matrix build stamps converge across chunked runs#227JoeMatt wants to merge 1 commit into
JoeMatt wants to merge 1 commit into
Conversation
The matrix stamps every row it writes with scripts/build-id.sh's output and, on resume, re-runs any row whose stamp differs from the current build. But the default results file (docs/cd-boot-matrix.md) is tracked, so writing the first row flipped the tree to "-dirty": the next invocation computed a different id, judged all the rows it had just written stale, and re-ran them. A chunked sweep (CD_MATRIX_MAX_RUNS) could therefore never settle, which is the one thing chunking exists for. Observed as 6 rows stamped 5a83d3d followed by "pass 1: 0/20 rows stamped 5a83d3d-dirty". Exclude the results file from the dirty check in build-id.sh, which both sides of the guard already share: gen-version-h.sh stamps the binary from the same script, so the binary's embedded id and the harnesses' VJ_EXPECT_BUILD stay in agreement. A source edit still yields "-dirty" and still invalidates earlier rows, so the stale-row guard keeps doing its job. Also warn when CD_MATRIX_OUT points at some other tracked file, which would hit the same non-convergence without being covered by the ignore list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
github-actions
Bot
added
tests
Regression:
|
| ROM | Status | Details | Diff |
|---|---|---|---|
| jagniccc | 🆕 NEW | no baseline | - |
| yarc | 🆕 NEW | no baseline | - |
| jagniccc (determinism) | ✅ PASS | identical across runs | - |
| yarc (determinism) | ✅ PASS | identical across runs | - |
| jagniccc (frameskip) | ✅ PASS | skip=0 matches skip=3 | - |
| yarc (frameskip) | ✅ PASS | skip=0 matches skip=3 | - |
| jagniccc (save state) | ✅ PASS | round-trip matches | - |
| yarc (save state) | ✅ PASS | round-trip matches | - |
| jagniccc (rewind) | ✅ PASS | rewind matches | - |
| yarc (rewind) | ✅ PASS | rewind matches | - |
Platform: Linux x86_64
Updated by CI at 2026-07-31T04:05:46.227Z
Regression:
|
| ROM | Status | Details | Diff |
|---|---|---|---|
| jagniccc | 🆕 NEW | no baseline | - |
| yarc | 🆕 NEW | no baseline | - |
| jagniccc (determinism) | ✅ PASS | identical across runs | - |
| yarc (determinism) | ✅ PASS | identical across runs | - |
| jagniccc (frameskip) | ✅ PASS | skip=0 matches skip=3 | - |
| yarc (frameskip) | ✅ PASS | skip=0 matches skip=3 | - |
| jagniccc (save state) | ✅ PASS | round-trip matches | - |
| yarc (save state) | ✅ PASS | round-trip matches | - |
| jagniccc (rewind) | ✅ PASS | rewind matches | - |
| yarc (rewind) | ✅ PASS | rewind matches | - |
Platform: Linux aarch64
Updated by CI at 2026-07-31T04:05:53.148Z
Regression:
|
| ROM | Status | Details | Diff |
|---|---|---|---|
| jagniccc | 🆕 NEW | no baseline | - |
| yarc | 🆕 NEW | no baseline | - |
| jagniccc (determinism) | ✅ PASS | identical across runs | - |
| yarc (determinism) | ✅ PASS | identical across runs | - |
| jagniccc (frameskip) | ✅ PASS | skip=0 matches skip=3 | - |
| yarc (frameskip) | ✅ PASS | skip=0 matches skip=3 | - |
| jagniccc (save state) | ✅ PASS | round-trip matches | - |
| yarc (save state) | ✅ PASS | round-trip matches | - |
| jagniccc (rewind) | ✅ PASS | rewind matches | - |
| yarc (rewind) | ✅ PASS | rewind matches | - |
Platform: Darwin arm64
Updated by CI at 2026-07-31T04:06:43.413Z
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a convergence issue in the CD boot-matrix tooling where chunked runs would continuously invalidate prior rows because writing the tracked results file made the working tree appear “dirty”, changing the build stamp across invocations.
Changes:
- Adds a
BUILD_ID_IGNOREmechanism toscripts/build-id.sh(and a--ignoresflag) so tracked results files don’t force a-dirtybuild id. - Updates
test/tools/cd_boot_matrix.shdocumentation and behavior to rely on the stable stamp across chunked invocations, and warns whenCD_MATRIX_OUTpoints at a tracked file not covered by the ignore list.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| test/tools/cd_boot_matrix.sh | Improves chunked resume behavior docs and adds a warning when the output file would cause non-converging stamps. |
| scripts/build-id.sh | Introduces an ignore list and --ignores output to prevent results-file writes from toggling the build id to -dirty. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+33
to
+37
| # Repo-root-relative paths, ONE PER LINE -- the list is fed to `grep -vxF`, | ||
| # which reads a newline-separated pattern set. Space-separating two entries | ||
| # makes a single literal pattern that matches no filename at all, silently | ||
| # restoring the non-convergence this list exists to prevent. | ||
| BUILD_ID_IGNORE='docs/cd-boot-matrix.md' |
Comment on lines
+125
to
+132
| if git ls-files --error-unmatch "$OUT" >/dev/null 2>&1 \ | ||
| && ! scripts/build-id.sh --ignores 2>/dev/null | grep -qxF "$OUT"; then | ||
| echo "WARNING: CD_MATRIX_OUT=$OUT is tracked by git but is not in" >&2 | ||
| echo " scripts/build-id.sh's ignore list. Writing rows will flip the build" >&2 | ||
| echo " id to '-dirty', so the next chunked invocation will treat every row" >&2 | ||
| echo " this one records as stale and re-run it. Add the path to" >&2 | ||
| echo " BUILD_ID_IGNORE, or point CD_MATRIX_OUT at an untracked file." >&2 | ||
| fi |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
test/tools/cd_boot_matrix.shstamps every row it writes withscripts/build-id.sh's output, and on resume re-runs any row whose stamp differs from the current build. That stale-row guard exists for a good reason — it is what stops an olddocs/cd-boot-matrix.mdfrom resurrecting ancient results as if they were fresh (the phantom "intermittent" Battle Morph? (pc_escape)row,final_pc=$8FBFB758, was exactly that).But the default results file is itself tracked, and
build-id.shappends-dirtywhen any tracked file is modified. So:abc1234and stamps its rows with it. Writing those rows dirties the tree.abc1234-dirty, sees every row invocation 1 recorded as "recorded by a different build", and re-runs all of them.Observed directly: a chunked sweep produced 6 rows stamped
5a83d3d, and the next invocation reportedpass 1: 0/20 rows stamped 5a83d3d-dirtyand started over from the top. Only a single unlimited invocation (CD_MATRIX_MAX_RUNS=0) from a clean tree could ever finish — which defeatsCD_MATRIX_MAX_RUNS, whose documented reason for existing is "lets a long sweep be chunked across short invocations".Fix
Exclude the results file from the dirty check via a new
BUILD_ID_IGNORElist inscripts/build-id.sh.The exclusion has to live in
build-id.shspecifically, not in the matrix script:gen-version-h.shstamps the binary from that same script, so an exclusion known only to the matrix would desync the binary's embedded id from the harnesses'VJ_EXPECT_BUILDand put the script into a rebuild loop that fails the build-identity guard on every run. One list, both sides agree.Implemented as name filtering rather than an
:(exclude)pathspec, so it needs no pathspec-magic support and a malformed entry can only fail toward-dirty(safe) instead of silently matching everything and disabling the guard.The guard is not weakened. A real source edit still yields
-dirtyand still invalidates earlier rows; only writes to the results file — which cannot change a byte of the binary — are ignored. The script also now warns ifCD_MATRIX_OUTpoints at some other tracked file, which hits the identical wall without being covered by the list.Verification
From a clean tree at this commit, with the core rebuilt to match (
v2.3.2 d8a7c79):already recorded by this build, skippingfor exactly the rows chunk 1 wrote, then advanced to the next titles. This is the behaviour the bug prevented.src/cd/cdrom.cflipped the stamp to-dirty, triggered the core rebuild, and made previously-recorded rows read asstale row (recorded by 'd8a7c79', current 'd8a7c79-dirty'). Stale-row resurrection stays impossible.-dirty; multi-entry ignore lists behave as documented.GAME_CODE 1/1on the hle row),git status --porcelainshowed onlydocs/cd-boot-matrix.mddirty. No other tracked file is written by a sweep.The matrix table was restored after each run, so no measured rows are included here — this is a tooling fix only, with no stage re-baseline.
🤖 Generated with Claude Code