fix(ci): prevent poisoned Rust caches

[wesbillman]

Summary

• roll every Swatinem/rust-cache use back from v2.9.2 to the last known-good v2.9.1 digest
• give Unit Tests a new sherpa-cache-v1 key so it cannot restore the existing poisoned artifact
• pin Renovate to v2.9.1 and add CI contracts that reject unsafe cache actions or a misplaced generation key

Why

After #5441 upgraded rust-cache to v2.9.2, warm-cache main Unit Tests runs began failing while linking buzz-voice with could not find native static library sherpa-onnx-c-api. The failed run at db5617dd1 restored the same 1.4 KB cache generation that had already failed at 01091c15a; the preceding cold run at 26f4c3ed3 downloaded sherpa 1.13.4 and passed.

v2.9.2 changed target cleanup, while sherpa-onnx-sys treats its prebuilt lib/ directory as proof that the native archive exists. Rolling back the action and invalidating the affected key removes both sides of that failure state without disabling target caching.

Validation

At 6da0037a0407fc498cd482fcbbb74c7a15907e9f:

• scripts/test-rust-cache-contract.sh
• scripts/test-rust-cache-contract-regressions.sh
• negative fixtures reject a bad digest in a newly named .yaml workflow and a generation key moved outside the cache action's with block
• YAML parse for all workflows
• release, desktop candidate, mobile release, mobile candidate, and mobile worktree source contracts
• just file-size-check
• pre-commit and pre-push hooks

The PR Unit Tests run proves the cold-cache path because pull requests restore but do not save Rust caches. The first successful main run after merge will save the new Unit Tests key; the following main run will exercise the warm restore.

Related issue

None found.

[wesbillman]

Carl, an automated reviewer, commenting via Wes’s GitHub account.

Blocking: make the contract discover every workflow. scripts/test-rust-cache-contract.sh hardcodes only ci.yml, mesh-lifecycle.yml, and release.yml. A future workflow can therefore add the known-bad v2.9.2 digest (or any other Swatinem/rust-cache pin) while this required contract still passes. I reproduced that at this head by adding .github/workflows/_rust-cache-contract-bypass.yml with Swatinem/rust-cache@6323deb...; the script printed rust cache contract passed and exited 0.

Please discover all .github/workflows/*.yml/*.yaml files (or otherwise use one authoritative inventory) before enforcing the digest set. The current rollback, Unit Tests generation bump, Renovate cap, and live cold-cache Unit Tests result otherwise look correct.

GitHub does not allow this account to request changes on its own PR, so this is posted as a comment review but remains blocking.

[jedwards27]

Verdict: REQUEST CHANGES

Reviewed: e23632941331502c0330e51d407e667bea26ef57..cb4ca303dad8e63515884803bda8792e84782b5f (exact live head cb4ca303dad8e63515884803bda8792e84782b5f)

Risk: Medium — this is CI/release-cache policy rather than product runtime behavior, but the regression guard is intended to prevent a known poisoned warm-cache state and currently admits two recurrence paths.

Blocking — the cache-safety contract is bypassable

scripts/test-rust-cache-contract.sh:5-9,21-23 hardcodes three workflow files even though the repository currently has 18 .github/workflows/*.yml/*.yaml files. I added a new workflow containing the known-bad v2.9.2 digest (6323deb...); the contract still printed rust cache contract passed and exited 0. A new or renamed workflow can therefore restore the poisoned action without the required gate noticing.

The generation-key check is also not structurally bound to the cache action: scripts/test-rust-cache-contract.sh:33-34 only looks for key: sherpa-cache-v1 anywhere in the textual unit-tests job. Removing the key from the Swatinem/rust-cache step and placing the same text in an unrelated step's env still passed. The gate can thus remain green while Unit Tests resumes restoring the poisoned cache generation.

Please discover every workflow (*.yml and *.yaml) and validate that all Swatinem/rust-cache uses carry the approved digest. Parse or otherwise inspect the Unit Tests action step structurally so its own with.key is sherpa-cache-v1. Add regression fixtures for a bad digest in a newly named workflow and for a key moved to another step.

The current source state itself is otherwise sound: all 10 current workflow uses are pinned to v2.9.1, Unit Tests currently attaches sherpa-cache-v1 to the right step, and the Renovate cap is narrow. No renderer, relay, schema, identity, persistence, or user-interface contract changed; product/UI and native validation are not applicable.

Validation at matching clean HEAD: baseline scripts/test-rust-cache-contract.sh passed; both mutations above incorrectly exited 0; all required GitHub checks are complete/successful except the path-correct Web skip. The Unit Tests PR job is cold-cache evidence only.

Residual risk: PR runs do not save this Rust cache, so warm restoration cannot be proven before merge. The first successful main run can populate the new key; only a subsequent main run can prove warm restore. That follow-up operational proof does not replace repairing the bypassable pre-merge contract.

[wesbillman]

Rebased onto current main at db5617dd1 and addressed both contract bypasses from review. The guard now discovers every .yml/.yaml workflow and structurally requires sherpa-cache-v1 on the Unit Tests rust-cache step; dedicated negative fixtures cover both recurrence paths.

[jedwards27]

:bot: Jude’s code review agent

Verdict: REQUEST CHANGES
Reviewed: db5617dd1541aeab7bacaf039b6ca98f856776d0..6da0037a0407fc498cd482fcbbb74c7a15907e9f (exact head 6da0037a0407fc498cd482fcbbb74c7a15907e9f)
Risk: medium — this PR defines the required guard against reintroducing a known-poisoning cache action.

Blocking finding

P2 — non-SHA Swatinem/rust-cache references bypass the new contract (scripts/test-rust-cache-contract.sh:26-47; regression coverage at scripts/test-rust-cache-contract-regressions.sh:34-63).

The contract discovers only uses whose ref is already exactly 40 hexadecimal characters, then validates those matches. Both independent review lanes added a new workflow using Swatinem/rust-cache@v2 / @v2.9.2; the contract printed rust cache contract passed and exited 0. A mutable tag, branch, or expression can therefore reintroduce the forbidden action while the required guard remains green.

Author action: detect every Swatinem/rust-cache@<ref> coordinate independently of ref shape, require the ref to equal the approved 40-character digest, and add a biting tagged/non-SHA regression fixture.

Verification owner: author/CI for the patch and regression; reviewer for exact-head mutation re-check.

Reconciliation and validation

The prior two findings are fixed: workflow discovery now covers top-level .yml and .yaml, and the Unit Tests generation key is structurally bound to the rust-cache step. Both lanes independently reproduced the remaining non-SHA bypass. Baseline contract and checked-in regression script passed; just file-size-check and git diff --check passed; all 10 current repository uses found by the review are pinned to the approved digest. Unit Tests was green when sampled; several other required checks were still running.

Warm-cache restoration remains post-merge main CI evidence. It is a confidence/operations gap, not an additional author blocker. No product/UI, relay, schema, identity, persistence, or native surface changed.

[jedwards27]

Verdict: request changes

Reviewed exact head 6da0037a0407fc498cd482fcbbb74c7a15907e9f against base db5617dd1541aeab7bacaf039b6ca98f856776d0.

P2 — the pin-enforcement contract ignores mutable/non-SHA rust-cache references

scripts/test-rust-cache-contract.sh:26-35 only recognizes Swatinem/rust-cache uses whose ref is already exactly 40 hexadecimal characters. Lines 37-47 validate only those recognized matches. Consequently, adding a workflow step such as:

- uses: Swatinem/rust-cache@v2

leaves the contract green: two independent fixture mutations at this head reproduced exit 0 with rust cache contract passed (using @v2 and @v2.9.2). The existing regression at scripts/test-rust-cache-contract-regressions.sh:34-43 exercises only a bad 40-character digest, so it does not catch this path.

This permits a future workflow to restore a mutable action reference while the required guard remains green, defeating the PR's safe-pin guarantee.

Required fix: detect every Swatinem/rust-cache@<ref> use independently of ref shape, then require the ref to equal the approved digest. Add a biting regression using a tag such as @v2 (preferably also an expression/non-SHA ref).

Author action: broaden detection and add the non-SHA regression.

Verification owner: author/CI for the patch and regression; reviewer for exact-head mutation re-check.

Re-checked and repaired

• Exhaustive top-level .github/workflows/*.{yml,yaml} discovery is now present at scripts/test-rust-cache-contract.sh:18-24.
• Unit Tests cache-key enforcement is scoped to the actual rust-cache step and its with.key at lines 49-94.
• Both checked-in contract scripts and just file-size-check passed in exact-head lane validation; all 10 current rust-cache uses found in the checked tree use the approved digest.
• The six-file diff has no product/UI, relay, schema, identity, persistence, or release-artifact behavior change. UI-state and accessibility validation are therefore not applicable; VISION.md is not in tension.

Residual risk: warm-cache restore remains post-merge main CI verification, not an author defect. Several unrelated required checks were still running when sampled; no completed required check was failing.

[wesbillman]

Addressed the latest non-SHA bypass review at af70f1e48 and rebased onto current main (a0298539f). The contract now detects every Swatinem/rust-cache@<ref> coordinate before validating the ref, so tags, branches, and expressions cannot evade the approved digest check. Added biting @v2 and ${{ matrix.cache-ref }} regression fixtures. Focused contracts, workflow YAML parse, file-size gate, pre-commit, and pre-push all pass.

[jedwards27]

:bot: Jude’s code review agent

Verdict: REQUEST CHANGES
Reviewed: a0298539f7043cd0f2d961030e60cc0fd82970b1..af70f1e48e1308ce846501277e1935102905b74b (exact live head af70f1e48e1308ce846501277e1935102905b74b)
Risk: medium — CI cache policy, with a known poisoned warm-cache failure mode.

P2 — valid YAML action-step forms bypass the cache-pin contract

scripts/test-rust-cache-contract.sh:26-35 discovers rust-cache actions with a line regex requiring the literal block-style key uses:. Valid workflow steps using another YAML spelling are invisible to it. In an isolated fixture at this exact head, both:

- uses : Swatinem/rust-cache@v2
- { uses: Swatinem/rust-cache@v2 }

parsed to mappings whose uses value is the mutable rust-cache ref, while BUZZ_RUST_CACHE_CONTRACT_ROOT=<fixture> scripts/test-rust-cache-contract.sh printed rust cache contract passed and exited 0. Independent replay with a flow mapping and a quoted "uses" key produced the same false pass. Because the repository's ordinary spellings still populate the script's uses list, its empty-list fallback cannot expose the omission.

A future valid workflow can therefore restore the known-bad or another unapproved action ref while the required safety gate remains green. The checked-in regressions at scripts/test-rust-cache-contract-regressions.sh:34-65 cover bad digest, tag, and expression only in the regex's preferred spelling, so they do not protect this path.

Author action: inspect workflow steps semantically with a YAML/workflow-aware parser and require every Swatinem/rust-cache uses value to equal the approved digest. Add biting regressions for at least a flow mapping and a non-canonical/quoted uses key.

Verification owner: author/CI for the patch and regressions; reviewer for exact-head mutation replay.

Reconciliation and validation

The earlier bypasses are repaired: top-level .yml/.yaml discovery is exhaustive, ordinary non-SHA refs are rejected, and Unit Tests' sherpa-cache-v1 is structurally bound to its rust-cache step. All 10 current runtime uses found in the tracked tree use e18b497… (v2.9.1), and the Unit Tests job succeeded on the cold path.

At matching clean HEAD: both checked-in contract scripts passed; the new YAML-spelling mutation incorrectly passed; CHECK_FILE_SIZES_BASE=a0298539f7043cd0f2d961030e60cc0fd82970b1 just file-size-check passed all core and desktop/web/mobile checks; git diff --check passed. All completed required GitHub checks were green when sampled; Web was path-correct skipped and Windows Rust remained in progress.

No product/UI, relay, identity, persistence, or native surface changed; VISION.md is not in tension and UI/native evidence is not applicable. Warm-cache restore remains post-merge main CI evidence—a CI-owned residual confidence gap, not an additional author blocker.

[wesbillman]

Replaced the textual workflow matcher at 0141689d3 with semantic YAML parsing and rebased onto current main (f6e6617a9). The contract now traverses parsed workflow mappings/arrays, validates every uses coordinate targeting Swatinem/rust-cache, and reads the Unit Tests cache step and with.key structurally. Added regressions for spaced keys, flow mappings, quoted keys, bad digests, mutable tags, expressions, and misplaced generation keys. I also adversarially replayed single-quoted flow mappings, YAML aliases, mixed-case coordinates, job-level reusable-workflow coordinates, and quoted/spaced keys; all were rejected. Focused contracts pass under both the Hermit shell and system Ruby 2.6, workflow YAML parses, file-size gate passes, and pre-commit/pre-push hooks pass.

[jedwards27]

:bot: Jude’s code review agent

Verdict: REQUEST CHANGES
Reviewed: f6e6617a9dcc2308d5039f8afaab974b49fb9577..0141689d379b3cad64792e7c3e75d4e83f39d204 (exact live head 0141689d379b3cad64792e7c3e75d4e83f39d204)
Risk: medium — this is the required recurrence guard for a known poisoned warm-cache failure.

P2 — local composite actions remain outside the cache-pin contract

scripts/test-rust-cache-contract.sh:16-18,46-57 parses only top-level .github/workflows/*.{yml,yaml}. It does not inspect local composite-action manifests (action.yml / action.yaml), whose runs.steps[*].uses coordinates execute actions through workflow-invoked wrappers.

I independently reproduced this at the exact clean head by adding an isolated .github/actions/cache/action.yml containing:

runs:
  using: composite
  steps:
    - { uses: Swatinem/rust-cache@v2 }

and a workflow step invoking ./.github/actions/cache. Ruby/Psych resolved the nested coordinate as Swatinem/rust-cache@v2, but BUZZ_RUST_CACHE_CONTRACT_ROOT=<fixture> scripts/test-rust-cache-contract.sh printed rust cache contract passed and exited 0. The checked-in regressions create only workflow files, so they do not protect this executable path.

A repository-local wrapper can therefore restore the forbidden mutable action while the required “every rust-cache use” guard stays green.

Author action: include local action.yml / action.yaml composite manifests in the semantic scan (at minimum repository-local actions reachable from workflows) and require every nested rust-cache coordinate to equal the approved digest. Add a biting composite-action fixture invoked by a workflow.

Verification owner: author/CI for the patch and fixture; reviewer for exact-head mutation replay.

Reconciliation and validation

The semantic YAML rewrite does repair every previously reported workflow-syntax bypass: new .yml/.yaml files, bad digests, mutable tags/expressions, spaced/quoted keys, flow mappings, and the moved Unit Tests generation key are now rejected. All 10 current workflow coordinates use e18b497…, and Unit Tests binds with.key: sherpa-cache-v1 to the cache step.

At matching clean HEAD: both checked-in contract scripts passed; independent quoted-flow/alias, case-variant, and missing-key mutations failed causally; the composite-action mutation incorrectly passed; CHECK_FILE_SIZES_BASE=f6e6617a9dcc2308d5039f8afaab974b49fb9577 just file-size-check and git diff --check passed. Unit Tests succeeded on the cold path. No completed required GitHub check was failing at final sample; some remained in progress.

No product/UI, relay, identity, persistence, or native surface changed; VISION.md is not in tension and UI/native evidence is not applicable. Warm-cache restore remains post-merge main CI-owned residual evidence, not another author blocker.

[jedwards27]

:bot: Jude’s code review agent

Verdict: REQUEST CHANGES
Reviewed: f6e6617a9dcc2308d5039f8afaab974b49fb9577..0141689d379b3cad64792e7c3e75d4e83f39d204 (exact head 0141689d379b3cad64792e7c3e75d4e83f39d204)
Risk: medium — required CI contract preventing poisoned/mutable Rust cache actions.

Finding

[P2] Local composite actions can reintroduce rust-cache while the required guard stays green. scripts/test-rust-cache-contract.sh:16-18,46-57 semantically walks only .github/workflows/*.{yml,yaml}. It does not inspect repository-local composite action manifests (action.yml / action.yaml), whose runs.steps[*].uses execute actions exactly like workflow steps.

An exact-head isolated fixture added .github/actions/cache/action.yml containing runs.using: composite and runs.steps: [{ uses: Swatinem/rust-cache@v2 }], then invoked it from a workflow via uses: ./.github/actions/cache. Ruby/Psych resolved the nested unsafe coordinate, but the contract printed rust cache contract passed and exited 0. The checked-in regression fixtures create workflow files only.

Consequence: a local cache wrapper can execute a forbidden mutable rust-cache ref while the safe-pin gate remains green, defeating the stated “every rust-cache use” recurrence contract.

Author action: include repository-local action.yml/action.yaml manifests in the semantic scan—at minimum .github/actions/**, preferably every tracked local action reachable from workflows—and require every nested rust-cache coordinate to equal the approved digest. Add a biting composite-action fixture invoked by a workflow.

Verification owner: author/CI for patch and fixture; reviewer/A Team for exact-head mutation replay.

Integrated evidence

The semantic YAML rewrite does repair all prior workflow-syntax bypasses: alternate uses spacing, flow mappings, quoted keys, new .yaml workflows, bad digests, mutable tags/expressions, and moved generation keys now fail causally. The Unit Tests key is structurally bound to the single rust-cache step. Independent mutations for those paths all failed with the intended diagnostics, and anchors/folded scalars were also rejected.

Baseline contract and checked-in regressions passed; explicit-base just file-size-check and git diff --check passed. The tree has 18 workflow files and ten direct rust-cache uses, all pinned to e18b497...; exact-head Unit Tests is green and logs show sherpa-cache-v1 on the cold path. No product/UI/runtime surface changed.

Residual risk: warm restore remains post-merge main CI evidence, not an additional author blocker. Any head movement invalidates this review.

[wesbillman]

Confirmed and fixed the composite-action bypass at 6226d8e69. The contract now follows reachable repository-local actions recursively, resolves action.yml before action.yaml exactly like the Actions runner, rejects path/symlink escapes, and includes regressions for an unsafe invoked composite action plus dual-manifest precedence. Focused contracts pass under both Ruby 4.0 and system Ruby 2.6; pre-push hooks pass. Requested exact-head re-review.

[jedwards27]

:bot: Jude’s code review agent

Verdict: APPROVE
Reviewed: f6e6617a9dcc2308d5039f8afaab974b49fb9577..6226d8e69c10f5f28461e256173be8ecaa9822b0 (exact head 6226d8e69c10f5f28461e256173be8ecaa9822b0)
Risk: medium — CI cache correctness and executable GitHub Actions reachability; no product/runtime surface changed.

The prior local-composite-action bypass is fixed. The contract now walks workflow-reachable local actions transitively, handles both action.yml and action.yaml, contains resolved paths under the contract root, and checks every discovered Swatinem/rust-cache coordinate against the approved v2.9.1 digest (scripts/test-rust-cache-contract.sh:16-27,29-92). The checked-in workflow→composite unsafe-ref regression is causal (scripts/test-rust-cache-contract-regressions.sh:106-139).

Independent replay established:

• The prior 0141689d… contract incorrectly accepted a workflow→local composite→Swatinem/rust-cache@v2 fixture.
• This head rejects that same fixture at the composite manifest.
• A nested workflow→action.yaml→action.yml unsafe reference is also rejected; replacing only the ref with e18b497… passes.
• Removing the recursive scan makes the checked-in regression fail.

Findings: no unresolved blocking or non-blocking code finding.
Author action: none.
Verification owner: CI/operator after merge for the first cache save and a subsequent warm restore on main; reviewer automation for exact-head check freshness.

Validation at matching clean HEAD: baseline contract PASS; full checked-in regression script PASS; CHECK_FILE_SIZES_BASE=f6e6617… just file-size-check PASS; git diff --check PASS. Repository scan found 18 workflow files and 10 production rust-cache coordinates, all pinned to e18b497…. Exact-head Unit Tests downloaded that digest, used sherpa-cache-v1, and passed. At the final lane polls, completed Unit Tests, Rust Lint, Security, Mobile, cross-compile, Desktop E2E Relay, and macOS build checks were green; remaining checks were still running with no completed failure.

Manual/native evidence: not applicable; no Desktop/web/mobile/native product code or UI changed.
Residual risk: PR runs do not save the cache, so post-merge save and subsequent warm restoration remain CI-owned evidence. Ruby/Psych semantics remain part of the guard’s trusted boundary; reviewed YAML forms and recursive local-action paths were mutation-proven.

[jedwards27]

:bot: Jude’s code review agent

Verdict: APPROVE
Reviewed: f6e6617a9dcc2308d5039f8afaab974b49fb9577..6226d8e69c10f5f28461e256173be8ecaa9822b0 (exact live head 6226d8e69c10f5f28461e256173be8ecaa9822b0)
Risk: medium — this is the required recurrence guard for a known poisoned warm-cache failure.

The previous local-composite-action blocker is resolved. The contract now semantically parses all 18 top-level workflow files, follows reachable repository-local actions transitively across action.yml and action.yaml, contains resolved paths under the contract root, and requires every discovered Swatinem/rust-cache coordinate to equal e18b497… (scripts/test-rust-cache-contract.sh:16-92). The Unit Tests job separately requires exactly one direct rust-cache step with with.key: sherpa-cache-v1 (:94-110).

Independent exact-head mutation replays established causality:

• the historical contract incorrectly passed a workflow → local composite → Swatinem/rust-cache@v2 fixture;
• the current contract rejects that fixture at the nested manifest;
• a workflow → action.yaml → action.yml transitive unsafe ref is also rejected, and replacing only that ref with the approved digest passes;
• deleting the recursive-scan implementation makes the checked-in regression fail;
• the safe action.yml / unsafe sibling action.yaml fixture confirms GitHub manifest precedence without falsely scanning the unreachable sibling (scripts/test-rust-cache-contract-regressions.sh:106-140).

At matching clean HEAD, both contract scripts passed, explicit-base CHECK_FILE_SIZES_BASE=f6e6617a9dcc2308d5039f8afaab974b49fb9577 just file-size-check passed all core plus Desktop/Web/Mobile gates, and git diff --check passed. A repository scan found ten production rust-cache coordinates, all on the approved digest. Exact-head Unit Tests job 97588651948 downloaded that digest, used sherpa-cache-v1 with PR save-if: false, and passed. At final submission freshness, head/base were unchanged and no completed GitHub check had failed; remaining checks were still running.

No product/UI, relay, identity, persistence, authorization, or native surface changed. VISION.md and TESTING.md introduce no conflicting requirement, and UI/native workflow evidence is not applicable.

Author action: none.
Verification owner: required CI for the remaining checks; CI/operator after merge for the first cache save and a subsequent warm restore on main.

Residual risk: PR execution proves the cold/lookup path but intentionally cannot prove post-merge save plus warm restoration. The contract relies on Ruby/Psych semantics for supported GitHub YAML forms; the relevant workflow and recursive local-action shapes were mutation-tested.