Train startup 2: a deferred island load reads nothing unseen, island-closed statements, the island's compile cache, build-time notes for guarded refusals, lazy initializers

.oxlintrc.jsonc

@@ -370,6 +370,8 @@
         "packages/compiler/src/frontend/lowering/**",
         "packages/compiler/src/backend/llvm/**",
         "packages/compiler/src/backend/immutability.test.ts",
+        "packages/compiler/src/backend/lazy-initializers.test.ts",
+        "packages/compiler/src/backend/lazy-initializers.ts",
         "packages/compiler/src/backend/loop-polls.test.ts",
         "packages/compiler/src/backend/may-throw.test.ts",
         "packages/compiler/src/backend/stack-observations.test.ts",

benchmarks/runtime/README.md

@@ -28,6 +28,7 @@ Application-shaped workloads for measuring the speed of compiled executables. Ea
 | `line-reader` | cpu | `readline` over a file stream: 'line' events for a 67 MB access log (every 50th line not ASCII), with a status count, a path tally and byte totals |
 | `log-lines` | cpu | 200k immediate `console.log`/`process.stdout.write` lines into the runner's pipe |
 | `log-summary`, `inventory-report` | cpu | the multi-module applications from `benchmarks/builds`, on 300k-line inputs |
+| `module-table-reads` | cpu | reads of a number table that another module builds at load (`benchmarks/runtime/module-table`): a hundred million through that module's one-line accessor from the entry's loop, and as many in a loop of the module's own. The table's module only computes, so its initializer runs at the first read; the workload is the cost of the check in front of a read |
 | `n-body` | cpu | five planets in an array of records advanced a million fixed steps by the pair loop over the upper triangle, then the system's total energy before and after |
 | `number-maps` | cpu | number-keyed Map/Set: a dense-id table, sparse ids in many small maps, get/has/set mixes |
 | `numeric-kernels` | cpu | sieve, hashing, `number[][]` matmul, `Float64Array` |
@@ -58,6 +59,7 @@ Application-shaped workloads for measuring the speed of compiled executables. Ea
 | `cli-config` | startup | a `deploy-plan` CLI: read a ~110 KB JSON manifest, validate, interpolate env values, order 80 services into dependency waves, print a summary |
 | `startup-islands-used`, `startup-islands-unused`, `startup-islands-cold` | startup | a command-line program with three dependencies in Node islands (`benchmarks/startup/islands`): an ES module package of about a hundred small modules, a CommonJS package whose load reads an environment variable that is normally unset, and a package it imports and does not use, which ships 1.3 MB of data files nothing loads (as most of a real package directory is). `-used` calls into the first two; `-unused` prints a version and uses none of them, so its time is what having islands costs a start that does not need them; `-cold` is `-used` on a first run, with an empty island module cache. All three need a libnode pack |
 | `startup-modules` | startup | the 240-module program in `benchmarks/startup/modules`: every module's top level runs (classes and instances, Maps and a shared registry filled at load, record tables, strings and regex literals, a regular expression built from a string, constant objects, closures) and the entry prints one line, so the time is module initialization |
+| `startup-unused-tables` | startup | the command-line tool in `benchmarks/startup/unused`, asked for its version: four modules build tables at load (number tables filled by loops, records derived from a smaller table, a catalogue flattened with `Object.entries` and `flatMap`, regular expressions assembled from strings) and the command reads none of them, so the time is what unused module-level tables cost a start |
 | `http-api` | server | a `node:http` JSON API: routing, query parsing, a seeded catalog, per-cart state, typed JSON bodies, `ETag`/`If-None-Match`, auth and request-id headers |
 
 Compare two checkouts (each with a built CLI and native artifacts for the host):

benchmarks/runtime/module-table/main.ts

@@ -0,0 +1,11 @@
+// Reads of a table that another module builds at load: through that module's
+// one-line accessor from a loop here, and in a loop of the module's own. The
+// table's module only computes, so its initializer may run at the first
+// read; whatever stands in front of a read then must not cost the loop.
+import { at, sum } from "./table.ts";
+
+const rounds = Number(process.argv[2] ?? "1");
+const n = rounds * 1_000_000;
+let total = 0;
+for (let i = 0; i < n; i++) total += at(i);
+console.log(total, sum(n));

benchmarks/runtime/module-table/table.ts

@@ -0,0 +1,16 @@
+// A table built when the module loads, and two ways to read it.
+function build(): number[] {
+  const out: number[] = [];
+  for (let i = 0; i < 1024; i++) out.push((i * 2654435761) % 1000);
+  return out;
+}
+export const TABLE: number[] = build();
+export const SCALE: number = TABLE.length / 512;
+export function at(i: number): number {
+  return TABLE[i & 1023]! * SCALE;
+}
+export function sum(n: number): number {
+  let total = 0;
+  for (let i = 0; i < n; i++) total += TABLE[i & 1023]! * SCALE;
+  return total;
+}

benchmarks/runtime/workloads/module-table-reads.json

@@ -0,0 +1,7 @@
+{
+  "entry": "module-table/main.ts",
+  "args": [
+    "100"
+  ],
+  "scaleArg": 0
+}

benchmarks/runtime/workloads/startup-unused-tables.json

@@ -0,0 +1,7 @@
+{
+  "entry": "../startup/unused/main.ts",
+  "launches": 20,
+  "args": [
+    "--version"
+  ]
+}

benchmarks/startup/README.md

@@ -18,3 +18,7 @@ Change the generator, not the files.
 ## islands/
 
 A command-line program with three dependencies in Node islands, for the `startup-islands-*` workloads; its row in `benchmarks/runtime/README.md` describes it. Those workloads need a libnode pack (`requires` in the same README).
+
+## unused/
+
+A command-line tool of four modules that build their tables when they load: number tables filled by loops, a table of records derived from a smaller one, a catalogue of groups flattened into rows with `Object.entries` and `flatMap`, and regular expressions assembled from strings. Its `--version`, which the `startup-unused-tables` workload runs, uses none of them, so the time to exit is what tables nobody reads cost a start. `check` uses each table once and prints one line.

benchmarks/startup/unused/colors.ts

@@ -0,0 +1,24 @@
+export interface Color {
+  name: string;
+  red: number;
+  green: number;
+  blue: number;
+  hex: string;
+}
+const HEX = "0123456789abcdef";
+function pair(value: number): string {
+  return HEX.charAt(value >> 4) + HEX.charAt(value & 15);
+}
+// A table of records derived at load from a smaller one.
+const BASE: Omit<Color, "hex">[] = [];
+for (let i = 0; i < 4096; i++)
+  BASE.push({ name: "c" + i, red: (i * 7) & 255, green: (i * 13) & 255, blue: (i * 29) & 255 });
+export const COLORS: Color[] = BASE.map((color) => ({
+  ...color,
+  hex: "#" + pair(color.red) + pair(color.green) + pair(color.blue),
+}));
+export const BY_NAME: Map<string, Color> = new Map();
+for (const color of COLORS) BY_NAME.set(color.name, color);
+export function hex(name: string): string {
+  return BY_NAME.get(name)?.hex ?? "";
+}

benchmarks/startup/unused/main.ts

@@ -0,0 +1,27 @@
+// A command-line tool whose modules build their tables when they load, and
+// whose `--version` uses none of them: the time to exit is what the unused
+// tables cost a start. `check` uses every table once.
+import { root, SQUARES } from "./squares.ts";
+import { hex, COLORS } from "./colors.ts";
+import { factor, UNITS } from "./units.ts";
+import { isVersion, PATTERNS } from "./patterns.ts";
+
+function check(): string {
+  return [
+    root(SQUARES[1234]!),
+    hex("c77"),
+    COLORS.length,
+    factor("u7_9"),
+    UNITS.length,
+    isVersion("1.20.3"),
+    Object.keys(PATTERNS).length,
+  ].join(" ");
+}
+
+const command = process.argv[2] ?? "--version";
+if (command === "--version") console.log("1.4.0");
+else if (command === "check") console.log(check());
+else {
+  console.error("unknown command: " + command);
+  process.exitCode = 2;
+}

benchmarks/startup/unused/patterns.ts

@@ -0,0 +1,16 @@
+// Regular expressions assembled from pieces at load.
+const NUMBER = "0|[1-9]\\d*";
+const WORD = "[a-zA-Z][a-zA-Z0-9-]*";
+const PIECES: Record<string, string> = {
+  version: `(${NUMBER})\\.(${NUMBER})\\.(${NUMBER})`,
+  tag: `(?:-(${WORD}(?:\\.${WORD})*))`,
+  range: `(<|>)?=?\\s*(${NUMBER})(?:\\.(${NUMBER}))?`,
+};
+export const PATTERNS: Record<string, RegExp> = {};
+for (const name of Object.keys(PIECES)) {
+  PATTERNS[name] = new RegExp("^" + PIECES[name] + "$");
+  PATTERNS[name + "Loose"] = new RegExp("[v=\\s]*" + PIECES[name], "i");
+}
+export function isVersion(text: string): boolean {
+  return PATTERNS["version"]!.test(text);
+}

benchmarks/startup/unused/squares.ts

@@ -0,0 +1,8 @@
+// Number tables filled by loops at load.
+export const SQUARES: number[] = [];
+for (let i = 0; i < 20000; i++) SQUARES.push(i * i);
+export const BY_ROOT: Map<number, number> = new Map();
+for (let i = 0; i < 4000; i++) BY_ROOT.set(SQUARES[i]!, i);
+export function root(square: number): number {
+  return BY_ROOT.get(square) ?? -1;
+}

benchmarks/startup/unused/units.ts

@@ -0,0 +1,22 @@
+// A catalogue reshaped at load: groups of entries flattened into rows.
+interface Unit {
+  factor: number;
+  symbol: string;
+}
+const CATALOG: Record<string, Record<string, Unit>> = {};
+for (let g = 0; g < 40; g++) {
+  const group: Record<string, Unit> = {};
+  for (let u = 0; u < 50; u++) group["u" + g + "_" + u] = { factor: g * 100 + u, symbol: "s" + u };
+  CATALOG["group" + g] = group;
+}
+export interface Row extends Unit {
+  id: string;
+  group: string;
+}
+export const UNITS: Row[] = Object.entries(CATALOG).flatMap(([group, units]) =>
+  Object.entries(units).map(([id, unit]): Row => ({ ...unit, id, group })),
+);
+export function factor(id: string): number {
+  for (const unit of UNITS) if (unit.id === id) return unit.factor;
+  return 0;
+}

docs/content/docs/coverage.mdx

@@ -289,6 +289,7 @@ Some code compiles but can behave differently than under Node.js. Coverage repor
 - `SC6002`: values of one record type are built in key orders scriptc cannot track per value, so the type keeps its declaration order, an object literal's key order differs from it, and the program observes the order through `JSON.stringify`, `Object.keys`, `Object.values`, `Object.entries`, `for...in`, or console output.
 - `SC6003`: `localeCompare` compares a string literal outside the built-in Latin-script collation.
 - `SC6004`: a call passes a class instance to a function that casts it to an unrelated class.
+- `SC6012`: a JavaScript statement that doesn't compile sits in the `try` block of a `try`/`catch` in the same function. It throws when it runs, so the `catch` clause runs even where the statement would have completed under Node.js, and the compiled program [doesn't report it at run time](/docs/dependencies).
 
 ## Gating builds and CI
 

docs/content/docs/dependencies.mdx

@@ -150,7 +150,9 @@ scriptc: this program reached code that scriptc could not compile; each such ope
 scriptc: not compiled: 'unescape' is part of the standard library types but has no scriptc lowering yet [SC2020]
 ```
 
-Set `SCRIPTC_DEFERRED_REPORT=0` in the program's environment to silence these lines. A program with no deferred operations never prints them. A library built with `scriptc build --lib` doesn't print them either: a deferred operation in a library throws its error, and nothing is written to the host process's stderr.
+Set `SCRIPTC_DEFERRED_REPORT=0` in the program's environment to silence these lines. A program with no deferred operations never prints them.
+
+A deferred operation that sits in the `try` block of a `try`/`catch` in the same function is different: the program wrote a handler for exactly that statement, as in `try { optional() } catch {}`. The program doesn't print anything for such a site when it runs, so its stderr stays what it is under Node.js. The build names the site instead, with warning `SC6012` at the statement: the `catch` clause runs even where the statement would have completed under Node.js, so the program's output can differ from there on. Set `SCRIPTC_DEFERRED_REPORT=all` to have the program report these sites at run time too. A site in a function that the `try` block calls, in a `catch` or `finally` block, or in a `try` that has only a `finally`, is reported at run time like any other. A library built with `scriptc build --lib` doesn't print them either: a deferred operation in a library throws its error, and nothing is written to the host process's stderr.
 
 ## Fix a package's types
 

docs/content/docs/how-it-works.mdx

@@ -184,6 +184,7 @@ Program objects define `main` and leave selected `scr_*` runtime functions undef
 - **Memory:** Values use reference counting. Acyclic values are freed when their last reference is released, and a cycle collector runs at defined collection points. When the compiler proves that a group of objects is released together, such as a tree that a recursive function builds and its caller only reads, the group is allocated in an arena and released at once, without freeing each object or collecting its cycles. Some cycles involving opaque native values can remain allocated. When the main thread's script ends, the process exits without freeing what the program still holds, and the operating system reclaims it; worker threads free their heaps when they exit. Builds with `--sanitize`, and processes run with `SCRIPTC_EXIT_FREE=1` (for external leak checkers), free everything at exit.
 - **Concurrency:** An `async` function or async generator that suspends only at its own `await` and `yield` expressions compiles to a stackless coroutine whose heap frame holds the values live across them; other suspending `async` functions and async generators use stackful fibers, as do debug builds, libraries, worker programs and Windows executables. The native event loop schedules microtasks, timers, and I/O, using kqueue on macOS and epoll on Linux.
 - **Networking:** Supported `net`, `http`, `https`, `tls`, `dgram`, and `dns` APIs use native implementations. TLS uses vendored mbedTLS.
+- **Node islands:** A package that runs in a Node island is loaded by Node.js inside the program's process, the first time the program uses it. The package's JavaScript travels in the executable and is extracted once into the user's cache directory, `$XDG_CACHE_HOME/scriptc/node-islands/<digest>/` or `~/.cache/scriptc/node-islands/<digest>/`, where the digest names the set of files. Beside it, in `<digest>.compile-cache/`, the program keeps Node.js's [module compile cache](https://nodejs.org/docs/latest-v24.x/api/module.html#module-compile-cache) for those files, so a later start loads the compiled code instead of compiling the source again. A program that runs the `typescript` package in an island started in 205 ms without the cache and in 102 ms with it; its first start, which installs the files and fills the cache, took 244 ms instead of 218 ms. A file that changed is compiled again, and a rebuilt program has directories of its own. `NODE_DISABLE_COMPILE_CACHE=1` runs a program without the cache, and `NODE_COMPILE_CACHE=<directory>` makes Node.js use a directory of your choice instead. A program that never starts its island creates neither directory.
 - **Numbers:** Values use JavaScript's double-precision representation. Number-to-string formatting uses shortest-roundtrip output and is tested against Node.js.
 - **Regular expressions and Unicode:** The runtime uses QuickJS's regular-expression and Unicode libraries, linked into programs that use them.
 

docs/content/docs/limitations.mdx

@@ -180,8 +180,12 @@ title: "Limitations"
 
 <include>../limitations/runtime-differences-from-node-js/event-order-across-native-code-and-islands.mdx</include>
 
+<include>../limitations/runtime-differences-from-node-js/island-packages-that-read-the-machine-when-they-load.mdx</include>
+
 <include>../limitations/runtime-differences-from-node-js/abort-listeners-across-native-code-and-islands.mdx</include>
 
+<include>../limitations/runtime-differences-from-node-js/compile-cache-in-node-islands.mdx</include>
+
 <include>../limitations/runtime-differences-from-node-js/child-process-event-order.mdx</include>
 
 ### Array bounds and runtime errors

docs/content/limitations/runtime-differences-from-node-js/compile-cache-in-node-islands.mdx

@@ -0,0 +1,6 @@
+- **Compile cache in Node islands:** Node.js keeps a module compile cache only when `NODE_COMPILE_CACHE` is set, and a compiled program keeps one by default for the JavaScript it runs in a Node island.
+  In island code `module.getCompileCacheDir()` therefore returns a directory where Node.js returns `undefined`, and `module.enableCompileCache()` reports that the cache is already enabled.
+  The cache is in the user's cache directory, beside the island's extracted files, and the program uses it only when that directory belongs to the user and no one else can write to it.
+  The program runs without the cache, and reports nothing, when the directory is not private in that way, cannot be created, or is on a file system with no free space, when the user has no home directory to keep it in, and when the process's effective user or group differs from its real one.
+  `NODE_DISABLE_COMPILE_CACHE=1` turns the cache off.
+  `NODE_COMPILE_CACHE=<directory>` makes Node.js use that directory as it does on its own, without those checks.

docs/content/limitations/runtime-differences-from-node-js/island-packages-that-read-the-machine-when-they-load.mdx

@@ -0,0 +1,6 @@
+- **Island packages that read the machine when they load:** The build keeps what an island package exports when it loads only when the load read nothing a run could answer differently.
+  A load that reads a clock, a random number, the process id, the arguments, the working directory, the user, the host, a file outside the package, or the time zone of a machine whose `TZ` is unset runs when the program imports the package, which starts Node.js there.
+  A load that reads an environment variable, the default locale, a time zone that `TZ` names, or whether a standard stream is a terminal waits, and runs at the import in a run where that answer differs from the build's.
+  A variable that has a value on the build machine keeps the load at the import unless the project lists it in `scriptc.nodeIslands.publicEnv`, so a build machine with `LANG` set does not defer a package that formats a number in the default locale.
+  A package marked `"evaluate": "lazy"` waits whatever its load reads, and the values it computed from the machine are then read from Node.js on first use.
+  The build cannot see three things a load can depend on, and keeps the build machine's answer for them: memory the package allocated without initializing it, how deep the stack is, and a value computed from the package's own path that no longer contains the path.