Skip to content

Evaluate comptime callbacks in one evaluator under both compilers - #3209

Closed
cramforce wants to merge 6 commits into
mainfrom
fix/sec-comptime-one-evaluator
Closed

cramforce wants to merge 6 commits into
mainfrom
fix/sec-comptime-one-evaluator

Conversation

@cramforce

@cramforce cramforce commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

What this fixes

Under the Node-hosted compiler (@scriptc/compiler's API, packages/cli/dist/bootstrap.js, this repository's tests and CI) a comptime callback ran with vm.runInNewContext inside the compiler's process. That is not a boundary. On main 827bc093d:

  • comptime(() => globalThis.constructor.constructor("return process")()) reaches the compiler's process: the built executable prints the compiler's pid and working directory. (() => 1).constructor("return this")() gets there too.
  • The lowerer called methods of the returned object: a result array with its own map receives one of the compiler's functions, and callback.constructor("return process")() reaches the host without touching a global.
  • Accessors on the result or on a thrown value, and promise reactions, ran in the compiler after the 2 s budget (a 6 s getter made a 7.4 s build that succeeded).
  • import("node:fs") and an unhandled Promise.reject(...) ended the compiler with an uncaught exception; an array with a hole crashed lowering (TypeError … reading 'kind'); there was no memory limit (4.3 GB peak inside the budget).

The native compiler evaluates callbacks in a QuickJS process and none of this applies to it; I built it from the same commit and ran the same programs against it. The docs described the native behaviour for both.

The change: one evaluator

Both compilers now run the same executable through the same driver, so a callback has one behaviour and one set of limits.

  • Evaluator. scriptc-comptime (QuickJS-ng plus comptime.c, now in native/llvm-codegen/comptime/) is built by each LLVM helper package's build:native and ships in @scriptc/llvm-<host>/bin beside the helper: the helper package is the one per-host package both compilers always resolve. The native CLI distribution's private copy (lib/scriptc-comptime), its build step and the toolchain manifest's comptime key are gone.
  • Driver. packages/compiler/src/frontend/comptime.ts (was native/comptime.ts): TypeScript's command line erases the types, the evaluator runs the JavaScript, and only tagged JSON comes back, so the compiler never calls or reads anything the callback built. frontend/comptime-node.ts and its TypeScript 5 use are deleted.
  • Runtime packs no longer carry libscriptc-quickjs.a, an archive no program linked; it existed for the native CLI's copy.

Options considered (a WebAssembly build of QuickJS in process, an N-API addon, V8 in node --permission, linking on first use) and why this one: the others leave two builds of the engine or two engines, and results must be identical under both compilers.

Limits and what is refused, the same under both compilers

  • Time: the existing 2,000 ms budget. A watchdog thread now ends an evaluation the engine does not interrupt (one long builtin call) half a second past it.
  • Memory: the existing 256 MB, now named: evaluation exceeded the 256 MB compile-time memory limit instead of the callback threw: out of memory.
  • Result: own enumerable properties, holes read as undefined, at most 1,000,000 elements per array, 512 levels, 64 MB; cycles and symbols are refused by name. (the callback threw: [uninitialized] and Command failed: …/scriptc-comptime … were what the native compiler printed for three of these.)
  • Asynchronous work left behind (a pending promise reaction, an unhandled rejection) is a build error. Natively it was dropped; Node itself exits 1 for the second.

Evaluator disagreements closed

After this change the corpus runs against the evaluator users get, which showed where it differed from Node. Measured on main, QuickJS against V8:

  • Math: sin, cos, tan, asin, acos, atan, atan2, sinh, cosh, tanh, asinh, acosh, atanh, exp, expm1, log, log1p, log2, log10, cbrt and hypot differed in the last digit for 0.03 % to 47 % of 4,000 inputs each (the C library against V8's fdlibm). The native compiler baked 4.57642774787696 where Node prints 4.576427747876961. The evaluator now calls the runtime's port of V8's file and V8's hypot; all 21 agree.
  • BigInt.asUintN: QuickJS-ng 0.15.1 answers -1n for asUintN(64, -1n) and for asUintN(64, 2n ** 64n - 1n). Corrected in the evaluator.
  • Refused by name, as a thrown error the build reports: toLocaleString, localeCompare, toLocaleLowerCase, toLocaleUpperCase, Date#toString, toTimeString, toLocale*String (no locale data), and Number#toString(radix) for a value that is not a safe integer (V8's digit generation).

No existing corpus program used an affected member: the gate shows every executable byte-identical.

Proof

All in Vercel Sandboxes (Intel Xeon @ 2.90GHz, 32 vCPU, Node 24.22.1, clang 22.1.8); the native compiler was built from the same commit as the Node-hosted one, on main and on this branch.

Escape attempts, each built with both compilers

Program main, Node host main, native this branch, both
globalThis.constructor.constructor(...) prints the compiler's pid no process no process
(() => 1).constructor("return this")() prints the compiler's pid no process no process
result array with its own map prints the compiler's pid no process no process
getter on the result that spins 6 s 7.4 s build, value baked budget error budget error, 2.1 s
promise reaction that spins 6 s runs in the compiler, 7.4 s dropped build error
import("node:fs") compiler dies a promise build error
unhandled rejection compiler dies baked build error
array with a hole internal compiler error type error at $[1] type error at $[1]
unbounded allocation 2.9 to 4.3 GB peak the callback threw: out of memory evaluation exceeded the 256 MB compile-time memory limit
table of Math.sin + cbrt + exp + hypot + atan2 Node's digits other digits Node's digits

Tests

  • packages/compiler/src/frontend/comptime.test.ts (14 tests): the driver and executable both compilers run. Constructor chains from every reachable prototype, result hooks, import(), require, blocking, the time limit through loops, patterns, one long builtin, accessors and thrown values, the memory limit on and off the heap, result rules, Math against Node's own Math on 600 inputs per member, BigInt.asUintN, the refusals, and one TypeScript run per file.
  • packages/compiler/test/comptime-isolation.test.ts (6 tests): the same attempts compiled by the Node-hosted compiler. Run against main, three fail on an assertion (the IR holds the host's answer), two never finish (the compiler is still inside the callback's loop minutes later), one is not an escape test.
  • tests/harness/self-hosting-native-driver.test.ts: the native compiler builds an escape program and the Math check, and builds and runs every comptime corpus program against Node.
  • Corpus: 8961-comptime-node-values.ts (a 48 by 21 table of Math members in full precision, FNV-1a hashes with BigInt.asUintN; fails natively on main) and 8962-comptime-result-data.ts. tests/diagnostics/comptime.ts gains eight cases; the nine existing diagnostics are unchanged.
  • The five existing comptime corpus programs print the same bytes as before under both compilers.

Zero cost. scripts/zero-cost-gate.mjs over benchmarks, the self-hosted compiler and the whole corpus (--corpus=all), no --allow, c64576d against base 4f6bb47: PASS (the commit after it changes one declaration in the macOS build of the evaluator). 5,237 programs are byte-identical, 102 fail to build on both sides, none differ. Compile time, all within tolerance: benchmarks −0.11 % [95 % CI −1.31 %, +0.87 %], self-compile −0.41 % [−2.93 %, +1.88 %], fixtures +0.52 % [−0.59 %, +1.64 %].

Compile time (build --emit=llvm, cache off, median of 11, interleaved):

Program Node host, main Node host, branch Native, main Native, branch
no comptime 1136 ms 1069 ms 67 ms 69 ms
1 call 1381 ms 1148 ms 137 ms 141 ms
10 calls 1515 ms 1397 ms 803 ms 162 ms
100 calls 1917 ms 2079 ms 7598 ms 393 ms

The Node host stops loading TypeScript 5 (235 ms once) and pays one tsc-ts run per file (45 ms) plus one evaluator process per call (1.6 ms). The native compiler paid a tsc-ts run per call. Run-to-run spread here is about 15 %, so the Node-host columns are equal within it; the native 10 and 100 call rows are real.

Lanes and bootstraps (97 sandbox jobs, on 75ad44a; the two commits after it change the evaluator, its build probe and tests, and the comptime unit, isolation, diagnostics and corpus tests were rerun on the head, plain and sanitized). Plain lane: 21,990 tests passed, 0 failed. Sanitized lane: 18,760 passed, 0 failed. The native bootstrap passes plain and sanitized (the seed, its command probes, the compiler rebuilding itself, the rebuilt compiler emitting itself). One suite did not run: packages/runtime/test/regex.test.ts timed out in its 60 s beforeAll, an ASan compile of runtime sources this change does not touch, in a shard of 370 files that held none of this change's tests; alone on the same tree it passes in 37 s.

Docs gate. NEXT_DIST_DIR=.next-check pnpm check in docs/: 25 of 25 route tests pass.

Docs

how-it-works.mdx said callbacks run "in an isolated evaluator inside the compiler". It now says what is true: a separate process that sees the ECMAScript built-ins and no object of the compiler or the host, with a time limit and a memory limit. The limitations page gains a "Build-time evaluation" section with the rules, limits and refusals, and says plainly that the evaluator is a process, not a hardened sandbox.

For reviewers

  • The helper packages gain a second executable. Touched for it: the seven packages/llvm-*/scripts/build.mjs, their package.json (bin, prepack) and notices, scripts/verify-llvm-package*.mjs, packages/cli/scripts/install-native.mjs, scripts/package-native-cli.mjs, scripts/verify-native-cli.mjs, the helper cache key in .github/actions/native-packages, and the mode restore in release.yml.
  • The evaluator is compiled with a clang-compatible driver, not CMake: clang on macOS and Linux (static on Linux, like the helper), Zig's Windows GNU target on Windows, where the helper itself is built with MSVC and the engine and the math port are not written for it. The Windows CI job installs Zig before the helper build for that.
  • Still open, listed in the limitations page where user-visible: two places in a result that hold the same object become two constants; Date() called as a function; engine error wording.

🤖 Generated with Claude Code

The Node-hosted compiler ran comptime callbacks with vm.runInNewContext in
its own process. A callback reached the compiler's process through the
context's constructor chain and through the result's own methods, which the
lowerer called with a compiler function; accessors and promise reactions ran
after the budget; an unhandled rejection or import() ended the compiler; and
there was no memory limit. The native compiler was not affected: it runs
callbacks in a QuickJS process.

Both compilers now start the same evaluator through the same driver:

- scriptc-comptime (QuickJS-ng and comptime.c, now under
  native/llvm-codegen/comptime) is built by each LLVM helper package and
  ships in its bin directory. The native CLI distribution's own copy, its
  build step and the toolchain manifest's comptime key are removed, and so
  is the QuickJS archive the runtime packs carried for it.
- frontend/comptime.ts replaces native/comptime.ts and
  frontend/comptime-node.ts. TypeScript's command line erases the types of
  all of a source file's callbacks in one run.
- The time limit is also enforced by a watchdog thread, the memory limit is
  enforced in the engine's allocator and reported by name, and results the
  evaluator refuses (cycles, symbols, size bounds) are named.
- Pending promise reactions and unhandled rejections are build errors.
- Math's transcendental members and hypot compute V8's values (the
  runtime's fdlibm port), BigInt.asUintN is corrected, and members whose
  Node result needs locale data, and toString with a radix for values that
  are not safe integers, throw by name instead of answering something else.

The docs state what the evaluator guarantees: a separate process with the
built-ins only and two limits, not a hardened sandbox.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Nov 7, 2026 11:43am UTC

Comment thread native/llvm-codegen/comptime/comptime.c Resolved
Comment thread native/llvm-codegen/comptime/comptime.c Resolved
The evaluator the Windows helper package builds with Zig answered the
probe correctly, with the carriage return a Windows executable writes
before the line feed. The probe compared the bytes exactly.

Co-Authored-By: Claude Code <noreply@anthropic.com>
…e result is read

A result that is a proxy over an array was written out as a record, so a
callback that returned one failed its array type where the Node-hosted
compiler used to accept it (Array.isArray looks through proxies). The
evaluator now applies the same rule.

The checks for pending promise reactions and unhandled rejections ran before
the result was written out. Accessors and proxy traps of the result run
while it is written, so work they queued passed unnoticed. The checks now
follow the encoding.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Apple's <malloc/malloc.h> does not compile when _POSIX_C_SOURCE is defined:
its typed-allocation declarations need a type the strict POSIX mode hides.
The evaluator defines _POSIX_C_SOURCE for clock_gettime and nanosleep and
needs one function from that header, so it declares it.

Co-Authored-By: Claude Code <noreply@anthropic.com>
One conflict: the node:fs import of packages/compiler/src/native/prepare.ts.
main dropped readFileSync from it when the link feature keys moved to one
list (#3245); this branch added existsSync for the evaluator's identity.
The import is now existsSync alone.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@cramforce

Copy link
Copy Markdown
Contributor Author

Merged main (19372cb) into the branch: 57363a9.

Conflict. One line, the node:fs import of packages/compiler/src/native/prepare.ts: main dropped readFileSync from it when the link feature keys moved to one list (#3245), and this branch added existsSync for the evaluator's identity. The import is now existsSync alone.

Re-run on the merged tree (Vercel Sandbox, Xeon 2.90GHz, 8 vCPU; the LLVM helper package rebuilt with scriptc-comptime beside it, the runtime pack rebuilt):

  • pnpm format:check, pnpm lint (the limitations partials included), pnpm test:ts7 --baselines-only: pass.
  • The seven comptime corpus programs, both backends: 14 passed plain, 14 passed sanitized.
  • comptime.test.ts and comptime-isolation.test.ts: 20 passed, plain and sanitized.
  • tests/harness/diagnostics.test.ts: 307 passed. tests/harness/llvm-package-mode.test.ts: 6 passed.
  • Unit tests (services, packages/compiler/src/native, startup, native-install, the Windows helper build test, code-quality, link-feature-lists, executable-cache): 173 of 174 passed.

The one failure is main's, not this branch's. packages/compiler/src/executable/executable-cache.test.ts › "every build and run option is in the cache keys or says why it need not be" fails on main itself since #3218 and #3260 both merged: --emit-lowered, --from-lowered and --from-lowered-accept have no entry in OPTION_KEY_RULES. CI here will show it until main has the three rules.

Not re-run: the zero-cost gate, the full lanes, the native bootstraps (self-hosting-native-driver.test.ts), native-cli-packaging.test.ts, the docs gate, and anything on macOS or Windows. The resolution is one import line; those results in the description are from before the merge.

main moved after the first merge: #3249 and #3251 put readFileSync back
into the node:fs import of packages/compiler/src/native/prepare.ts (the
build cache reads its recorded inputs there), where this branch has
existsSync. The import now names both.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@cramforce

Copy link
Copy Markdown
Contributor Author

main moved after the first merge and the branch conflicted again in the same line. Merged main (50bcfe1): d00f8ec.

Conflict. #3249 and #3251 put readFileSync back into the node:fs import of packages/compiler/src/native/prepare.ts (the build cache reads its recorded inputs there), where this branch has existsSync. The import names both now; nothing else conflicted, though main changed 136 other lines of that file around the evaluator's identity, so the checks ran again.

Re-run on this tree (Vercel Sandbox, Xeon 2.90GHz, 8 vCPU; helper package, scriptc-comptime and runtime pack rebuilt): pnpm format:check, pnpm lint, pnpm test:ts7 --baselines-only pass; the seven comptime corpus programs on both backends, 14 passed plain and 14 sanitized; comptime.test.ts and comptime-isolation.test.ts, 20 passed plain and 20 sanitized; diagnostics.test.ts, 309 passed; llvm-package-mode.test.ts, 6 passed; the unit set (now with prepare.test.ts's cache contracts), 206 of 207 passed. The one failure is executable-cache.test.ts › "every build and run option is in the cache keys…", which is main's (#3306 has the fix).

Not re-run: the zero-cost gate, the full lanes, the native bootstraps, native-cli-packaging.test.ts, the docs gate, macOS and Windows.

@cramforce

Copy link
Copy Markdown
Contributor Author

Superseded by train security-1, #3430. The train carries this branch at d00f8ec, merged with current main (three conflicts, all in CI files and the test cost table; the resolutions are in the merge commit's message), and adds the two patterns of the evaluator's inputs to the runtime pack's cache key, which a test main gained after this branch was cut requires. Validation of the merged tree is in #3430's description.

🤖 Generated with Claude Code

@cramforce cramforce closed this Nov 8, 2026
yowainwright pushed a commit to yowainwright/scriptc that referenced this pull request Nov 9, 2026
Compile-time callbacks run in one evaluator process under both compilers
(vercel-labs#3209).

Conflicts, resolved by reading both sides:

- .github/actions/native-packages/action.yml: the helper cache key keeps the
  branch's two added inputs (the evaluator's engine sources and the math
  port), and main's comment on the runtime pack's key stays below it.
- .github/workflows/ci.yml, the Windows job: Zig is installed before the
  helper build, which now builds the evaluator with it, and main's restore
  of ICU's archives stays before the runtime pack build. Zig is installed
  once.
- scripts/ci-test-costs.json: main's table, with the branch's rename
  applied: the evaluator's driver test is listed under its new path with
  main's measured cost, and the deleted test's rows are gone.

Co-Authored-By: Claude Code <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — d00f8ec7 Deployed Nov 7, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant