Skip to content

Train security-1: one compile-time evaluator under both compilers; toolchain downloads pinned by SHA-256 and actions by commit - #3430

Merged
cramforce merged 14 commits into
mainfrom
train/security-1
Nov 8, 2026
Merged

cramforce merged 14 commits into
mainfrom
train/security-1

Conversation

@cramforce

Copy link
Copy Markdown
Contributor

Two security changes that were open as their own pull requests, merged as one train on current main. Each was already reviewed by the bot on its own pull request; every thread there is resolved.

What lands

  • Evaluate comptime callbacks in one evaluator under both compilers #3209: Evaluate comptime callbacks in one evaluator under both compilers. Under the Node-hosted compiler, a comptime callback ran inside the compiler's own process. Code that runs at build time was therefore not separated from the compiler and the machine it runs on, had no memory limit, and could keep running past its time budget; the native compiler already ran callbacks in a separate evaluator process, so the two compilers could also bake different values. Now both compilers run the same evaluator executable through the same driver: a separate process that sees the ECMAScript built-ins and nothing of the compiler or the host, with one time limit, one memory limit and one set of rules for results, and only data comes back. The evaluator ships in the per-host LLVM helper package; the runtime packs no longer carry an engine archive that no program linked. The docs describe it as it is: a process with limits, not a hardened sandbox.
  • Pin toolchain downloads by SHA-256 and GitHub Actions by commit #3185: Pin toolchain downloads by SHA-256 and GitHub Actions by commit. CI, the release jobs and the Sandbox image downloaded toolchain archives (LLVM, Node, Zig, the LLVM apt key) without a check, or checked them against a list fetched from the same place as the archive, and referenced actions by tags that can move. Now every such download is checked against a digest committed in scripts/toolchain.sha256 before it is unpacked, every uses: names a commit with its version in a comment, and ci.yml and release.yml limit the default token to reading the repository.

Not in this train: #1197 (compiler protections for the C runtime packs). It changes every executable and costs run time, so it gets its own decision with measurements.

What the train adds on top of the two branches

  • Main gained eleven uses: lines by tag after Pin toolchain downloads by SHA-256 and GitHub Actions by commit #3185 was cut (the ICU download cache action, two cache steps of the native packages action, the generative workflow, one job of ci.yml, one step of release.yml). They are pinned to the same commits as the rest. All 160 uses: lines that name an action name a commit; for each of the seven actions the version tag names that commit on GitHub today.
  • Main's retry settings for the LLVM downloads (five attempts, thirty seconds apart) are kept inside the branch's download-then-check steps.
  • tests/harness/ci-supply-chain.test.ts now asks for one digest check per download line, not one per file, and has a test for the ICU download cache that main added: both restore steps key the cache by the file that holds ICU's pins, and a reader given a cache directory whose archives have other bytes refuses them, refuses a download with other bytes too, and unpacks nothing. (ICU's archives were already checked against their pins on every use; the test pins that the cache is no way around it.)
  • The runtime pack's cache key names the evaluator's two inputs, as the helper's key does (tests/harness/native-packages-cache.test.ts, which main gained after Evaluate comptime callbacks in one evaluator under both compilers #3209 was cut, requires it).
  • Zig is installed once in the Windows job, before the helper build that now needs it; main's restore of ICU's archives stays before the pack build.

Validation

All in Vercel Sandboxes (Ubuntu x86-64, Node 24.22.1, clang 22.1.8).

Full round on the train merged with main 59950fefd3 (the two branches, the train's own commits; the head of this pull request adds one more merge of main, checked below):

  • Type check: no TypeScript errors.
  • Lanes: plain 25,182 passed, 0 failed; sanitized 21,747 passed, 0 failed.
  • Native bootstraps: the plain one (the seed's command probes, the compiler rebuilding itself) and the sanitized one (command probes, the rebuilt compiler emitting itself, the rebuilt frontend) both ran to the end.
  • Zero-cost gate over the benchmarks, the self-hosted compiler and the whole corpus, no allow list: 5,822 executables byte-identical, 328 fail to build on both sides, none differ. One row is a program Evaluate comptime callbacks in one evaluator under both compilers #3209 adds (corpus/8962-comptime-result-data.ts): main's compiler stops on it with an internal error, the train compiles it.
  • Compile time, all within tolerance: benchmarks +0.09 % [95 % CI −1.72 %, +1.51 %], self-compile +0.76 % [−0.09 %, +1.36 %], fixtures +0.36 % [−0.70 %, +1.43 %].

Round before that, on main 6642a17b5f (the same train commits): bench-runtime, 73 workloads: geomean −0.4 %, 72 neutral, 1 faster, 0 slower; Test262 (Object, Reflect, JSON, Proxy, Function): 0 regressed. Its lanes showed one real failure, which is fixed here (the runtime pack's cache key, last bullet of the section above), and five tests that ran into their time limits on loaded boxes; all of them pass in the round above, and the two that were rerun alone take the same time on main's tree as on the train's (148.9 s and 147.7 s).

The head of this pull request is that tree plus a merge of main 5295319e16 (no conflict; one file changed on both sides, in different places): the gate against that main shows 5,827 of 6,158 executables byte-identical, 330 that fail to build on both sides, and none that differ, and the tree type-checks. The gate's compile-time shards on this head were still running at the push; their result follows as a comment. Lint, format, the workflow, cache-key and pin tests, the evaluator's tests and main's guard tests pass on it, and the checks of "The pins fail closed" below were repeated on it.

The pins fail closed

Run on the train's tree:

  • The verifier accepts the real llvm-snapshot.gpg.key; with one digit of its pin changed it exits 1 (has SHA-256 8b2a…, pinned 0b2a…); with one byte appended to the file it exits 1; for a name with no pin it exits 1 (pins no SHA-256 for …).
  • scripts/install-zig.sh 0.16.0 with a wrong pin for Zig stops at the check with exit 1 and leaves the install directory untouched.
  • The Download LLVM step of .github/actions/llvm-linux, run as written with a pin file whose LLVM digest has one digit changed: it downloads the 1.80 GB archive, exits 1 at the check, and unpacks nothing. The same file against the committed pin matches.
  • The ICU download cache: a cache directory with the two real archives is used without a request; one with files of the same names and other bytes is refused by both readers.
  • Every workflow and action file parses (15 files, 221 uses: lines). actionlint 1.7.7 reports the same three findings as on main, none new.

Not exercised

  • release.yml's download steps and its top-level permissions only run in a release; llvm-dist.yml is manual; the docker build of Dockerfile.sandbox was not run.
  • The macOS and Windows jobs run for the first time with the pins and with the evaluator in this pull request's own CI.
  • The first run after this lands downloads LLVM again (now checked) and rebuilds the helper and pack caches once, because their keys include the changed files.

Supersedes #3209 and #3185.

🤖 Generated with Claude Code

cramforce and others added 14 commits November 6, 2026 21:49
LLVM archives were downloaded without a checksum in CI, in the release jobs
and in the LLVM distribution workflow; Node and Zig for the Sandbox image
were checked against a checksum list fetched from the same origin as the
archive.

- scripts/toolchain.sha256 holds the SHA-256 of every toolchain archive that
  CI, the release jobs and the Sandbox image download (LLVM for Linux x64 and
  arm64 and for Windows, the LLVM sources, Node, Zig, the LLVM apt key), and
  scripts/verify-toolchain.sh checks a file against it. Every download runs
  the check before the file is unpacked; a different digest or a file with no
  pin fails.
- The Linux LLVM cache key is the hash of the pin file, so a cache never
  outlives the digest its archive was checked against.
- Every action is referenced by commit with its version in a comment, and
  Dependabot proposes the updates once a month.
- ci.yml and release.yml get top-level permissions (contents: read); the
  jobs that publish, read other runs' artifacts or write the GitHub release
  already declare what they need.
- tests/harness/ci-supply-chain.test.ts enforces the three rules.
- .dockerignore lets the Sandbox image build see the three scripts its
  Dockerfile copies.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The seed contract test found the upload and download steps by their tag
(actions/upload-artifact@v4). The actions are pinned to commits now, so the
lookup returned undefined and the test threw. Select them by name, as the test
plan's test already does.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The Node-hosted compiler ran comptime callbacks with vm.runInNewContext in
its own process. A callback reached the compiler's process through the
context's constructor chain and through the result's own methods, which the
lowerer called with a compiler function; accessors and promise reactions ran
after the budget; an unhandled rejection or import() ended the compiler; and
there was no memory limit. The native compiler was not affected: it runs
callbacks in a QuickJS process.

Both compilers now start the same evaluator through the same driver:

- scriptc-comptime (QuickJS-ng and comptime.c, now under
  native/llvm-codegen/comptime) is built by each LLVM helper package and
  ships in its bin directory. The native CLI distribution's own copy, its
  build step and the toolchain manifest's comptime key are removed, and so
  is the QuickJS archive the runtime packs carried for it.
- frontend/comptime.ts replaces native/comptime.ts and
  frontend/comptime-node.ts. TypeScript's command line erases the types of
  all of a source file's callbacks in one run.
- The time limit is also enforced by a watchdog thread, the memory limit is
  enforced in the engine's allocator and reported by name, and results the
  evaluator refuses (cycles, symbols, size bounds) are named.
- Pending promise reactions and unhandled rejections are build errors.
- Math's transcendental members and hypot compute V8's values (the
  runtime's fdlibm port), BigInt.asUintN is corrected, and members whose
  Node result needs locale data, and toString with a radix for values that
  are not safe integers, throw by name instead of answering something else.

The docs state what the evaluator guarantees: a separate process with the
built-ins only and two limits, not a hardened sandbox.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The evaluator the Windows helper package builds with Zig answered the
probe correctly, with the carriage return a Windows executable writes
before the line feed. The probe compared the bytes exactly.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e result is read

A result that is a proxy over an array was written out as a record, so a
callback that returned one failed its array type where the Node-hosted
compiler used to accept it (Array.isArray looks through proxies). The
evaluator now applies the same rule.

The checks for pending promise reactions and unhandled rejections ran before
the result was written out. Accessors and proxy traps of the result run
while it is written, so work they queued passed unnoticed. The checks now
follow the encoding.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Apple's <malloc/malloc.h> does not compile when _POSIX_C_SOURCE is defined:
its typed-allocation declarations need a type the strict POSIX mode hides.
The evaluator defines _POSIX_C_SOURCE for clock_gettime and nanosleep and
needs one function from that header, so it declares it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
One conflict: the node:fs import of packages/compiler/src/native/prepare.ts.
main dropped readFileSync from it when the link feature keys moved to one
list (#3245); this branch added existsSync for the evaluator's identity.
The import is now existsSync alone.

Co-Authored-By: Claude Code <noreply@anthropic.com>
main moved after the first merge: #3249 and #3251 put readFileSync back
into the node:fs import of packages/compiler/src/native/prepare.ts (the
build cache reads its recorded inputs there), where this branch has
existsSync. The import now names both.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Compile-time callbacks run in one evaluator process under both compilers
(#3209).

Conflicts, resolved by reading both sides:

- .github/actions/native-packages/action.yml: the helper cache key keeps the
  branch's two added inputs (the evaluator's engine sources and the math
  port), and main's comment on the runtime pack's key stays below it.
- .github/workflows/ci.yml, the Windows job: Zig is installed before the
  helper build, which now builds the evaluator with it, and main's restore
  of ICU's archives stays before the runtime pack build. Zig is installed
  once.
- scripts/ci-test-costs.json: main's table, with the branch's rename
  applied: the evaluator's driver test is listed under its new path with
  main's measured cost, and the deleted test's rows are gone.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Toolchain downloads are checked against pinned SHA-256 digests and actions
are referenced by commit (#3185).

Conflicts, resolved by reading both sides:

- .github/actions/native-packages/action.yml: the helper cache key from the
  train (with the evaluator's inputs), main's comment, and the branch's
  commit pin on the cache step below it.
- .github/workflows/ci.yml and release.yml, the Windows LLVM steps: the
  branch's layout stays (one step downloads and checks the archive, the next
  unpacks it), so main's download line inside the unpack step is gone; the
  download keeps main's five attempts thirty seconds apart.
- .github/workflows/llvm-dist.yml and release.yml, the Linux LLVM steps: the
  branch's check after each download, with main's retry settings.

Main gained eleven `uses:` lines by tag after the branch was cut (the ICU
download cache action, two cache steps of the native packages action, the
generative workflow, one job of ci.yml and one step of release.yml). They
are pinned here to the same commits as the rest; each of the seven pinned
actions was checked against GitHub: its version tag names that commit.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The supply-chain test asked for one digest check per file that downloads a
toolchain archive. It now asks for one per download line.

ICU's release archives are downloaded by the pack build, and CI keeps them
in a cache that several jobs restore. A new test pins that the cache is no
way around the digests: both restore steps key the cache by the file that
holds ICU's pins, and a reader given a cache directory whose archives have
other bytes refuses them, refuses a download with other bytes too, and
unpacks nothing.

Co-Authored-By: Claude Code <noreply@anthropic.com>
The LLVM helper's cache key gained two inputs with the compile-time
evaluator (the engine's sources and the math port). A runtime pack is
rebuilt whenever its helper is, so the pack's key lists every pattern the
helper's key lists, and tests/harness/native-packages-cache.test.ts, which
main gained after the evaluator's branch was cut, checks exactly that. The
two patterns are added to the pack's key. Its broader patterns already
covered the same files, so no pack is built from other inputs than before.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Main gained train lang-5 (#3422). No conflict, and no file that both sides
changed: main's 44 files are compiler sources, tests and test programs; the
train's are the evaluator, the pack and helper builds, and CI.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Main gained train cli-14 (#3397) and the program host fixture's check
(#3428). No conflict. One file changed on both sides, the lowerer: main's
change is outside the lines the evaluator's branch touches. Nothing under
.github changed on main.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Nov 8, 2026 3:55pm UTC

@cramforce

Copy link
Copy Markdown
Contributor Author

The gate round on this head (b0d018e against main 5295319) has ended: PASS.

  • 5,830 executables byte-identical, 330 fail to build on both sides, none differ; the one other row is the program Evaluate comptime callbacks in one evaluator under both compilers #3209 adds (corpus/8962-comptime-result-data.ts), which main's compiler cannot build.
  • Compile time, all within tolerance: benchmarks +0.49 % [95 % CI −0.65 %, +1.99 %], self-compile −1.55 % [−3.15 %, +0.36 %], fixtures +0.06 % [−0.40 %, +0.51 %].
  • The tree type-checks, and the self-hosted compiler builds its own source.

Lanes and both native bootstraps are the full round on the tree one merge of main earlier, as the description says (0 failed).

🤖 Generated with Claude Code

Comment thread scripts/verify-toolchain.sh Resolved
@cramforce

Copy link
Copy Markdown
Contributor Author

CI on this head (b0d018e), by cause, at 16:48 UTC.

Green: lint and format; every Linux lane shard (plain 20 of 20, sanitized 20 of 20, Effect 16 and 16, Node island conformance 4 of 4, the native frontend jobs, three.js); the Linux, macOS, Windows, Android and iOS builds; the four libnode packs; the bootstrap seeds; TypeScript 7 parity and the tsgo jobs; fuzz; the Windows jobs. Of the Linux native bootstrap jobs, the plain commands job has passed; the plain rebuild and the sanitized rebuild-emit and rebuild-frontend jobs were still running. Both bootstraps ran to the end in the Sandbox round on the tree one merge of main earlier (see the description).

Red: four macOS jobs, one cause, which is not this change.

  • test (macOS x64 LLVM artifacts, no clang) and test (macOS arm64 LLVM artifacts, no clang, 1/5): tests/harness/node-island-load-reads.test.ts, "a load that reads a clock evaluates where it is imported".
  • test (macOS, plain, darwin 3/3) and test (macOS arm64 LLVM artifacts, no clang, 5/5): the program 9811-node-island-load-reads-clock/main.cjs in the differential corpus.

The same test and the same program are red in the same jobs on other pull requests' runs today (train/pi-1b, train/islands-5, train/cli-22); main's own push runs do not run the macOS jobs. Every pinned step of the four jobs passed before the tests started: their logs hold no digest mismatch and no unresolved action. Two macOS sanitized shards and one arm64 shard were still running.

The pins in the jobs they touch:

  • Windows (test (Windows x64 MSVC LLVM artifacts), passed): verify-toolchain: clang+llvm-22.1.8-x86_64-pc-windows-msvc.tar.xz matches its pinned SHA-256; the ICU download cache was restored through restore-keys by the pinned cache action, and the runtime pack build that checks the restored archives against their pins passed.
  • macOS: the ICU download cache was restored from its primary key, and build (macOS arm64 native packages), which now also builds the evaluator, passed.

Review: one bot thread (line endings of the verifier on Windows): answered with the Windows job's output and resolved. No finding in a review body.

🤖 Generated with Claude Code

@cramforce

Copy link
Copy Markdown
Contributor Author

Addition to the CI summary above (16:52 UTC): a fifth macOS job has ended red, test (macOS, san, darwin 3/3), on the same program as the others, 9811-node-island-load-reads-clock/main.cjs (sanitized differential corpus). No digest mismatch and no unresolved action in its log. Still running: one macOS sanitized shard and three Linux native bootstrap jobs.

🤖 Generated with Claude Code

@cramforce
cramforce merged commit 72c4326 into main Nov 8, 2026
127 of 132 checks passed
@cramforce
cramforce deleted the train/security-1 branch November 8, 2026 16:57

This branch was successfully deployed

1 active deployment
Preview — b0d018e0 Deployed Nov 8, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant