Record fs.lstatSync/statSync kind as payload-free stat records

doc/file-formats.md

@@ -145,7 +145,11 @@ into the lockfile/bundle `config` block (`debug`, `childProcess`, `fs` and
   what distinguishes a resource from code per file, and tells a reader how to
   decode the bundle payload. Filesystem captures (`stasis run --fs=sync`) add
   `directory` for a `fs.readdirSync` listing (the content is a sorted JSON array
-  of names); a captured `fs.readFileSync` reuses the code or resource tags above.
+  of names) and the **payload-free** stat records `stat:file` / `stat:directory`
+  for an `fs.lstatSync`/`fs.statSync` of a path that was never read, listed, or
+  imported — those attest only that the path existed with that kind, so they
+  appear *only* here in `formats`, never in a module's hashed `files`; a captured
+  `fs.readFileSync` reuses the code or resource tags above.
   The integrity of a `directory` listing is the sha512 of its JSON text, just
   like any other content. The loader picks module-vs-commonjs and (for the
   `*-typescript` formats) whether type-shaped syntax is stripped purely from
@@ -214,7 +218,10 @@ reporting failures) still persists what it cleanly captured.
   Resource files use `resource` (raw UTF-8 payload) or `resource:base64`
   (base64 payload) — the per-file tag that both marks a file as a resource
   and says how to decode its bundle bytes. A `directory` tag (a `stasis run
-  --fs` capture) marks a raw-UTF-8 JSON listing. May be missing per file for code
+  --fs` capture) marks a raw-UTF-8 JSON listing; a `stat:file`/`stat:directory`
+  tag (an `--fs` `lstatSync`/`statSync` capture) is payload-free — the tagged
+  path has no `files` entry at all, the format alone attests its existence and
+  kind. May be missing per file for code
   whose format Node infers. TypeScript sources are stored verbatim (types
   intact); Node strips the types at load time based on the format.
 - `imports`: conditions → parent file → specifier → resolved
@@ -343,12 +350,14 @@ a frozen run verifies a copied asset byte-for-byte just as it does code.
 ## Filesystem captures (`stasis run --fs=sync` / `--fs=async`)
 
 The loader hooks capture the module graph; `--fs=sync` additionally monkey-patches the
-**sync** readers `fs.readFileSync` and `fs.readdirSync`, plus the metadata/existence probes
-`fs.lstatSync`/`fs.statSync`/`fs.existsSync`/`fs.accessSync`/`fs.realpathSync` (and only
+**sync** readers `fs.readFileSync` and `fs.readdirSync`, the kind-recording stat calls
+`fs.lstatSync`/`fs.statSync`, plus the existence probes
+`fs.existsSync`/`fs.accessSync`/`fs.realpathSync` (and only
 those — no streams, `fs.opendir`, `fs.readlink` (a resolver probe that already treats a read
 error as "not a symlink", so an absent recorded file resolves correctly unserved), the
 deprecated callback `fs.exists`, …) so a program's
-explicit file reads are recorded into the bundle (`--bundle=add|replace`) and served back
+explicit file reads — and the kind (file vs directory) of each path it stats — are
+recorded into the bundle (`--bundle=add|replace`) and served back
 from it (`--bundle=load`). The same `--fs=…` flag is needed on the load run for the patch
 to serve; an un-captured read falls through to the real disk read. The mode can equivalently
 be set as `"fs": "sync" | "async"` in `stasis.config.json` (or `EXODUS_STASIS_FS`), which
@@ -381,9 +390,21 @@ Captures live in the usual `sources`/`modules` buckets, tagged in `formats`:
   is not captured and is not served from the bundle on load. A listing captured
   *at* a package bucket root sits at the bucket's own key: rel `''` inside its
   `files`, format keyed at the bucket dir — `.` for the project root itself.
-- `fs.lstatSync(path)` and `fs.statSync(path)` are not themselves recorded; on a load
-  run, for a path the bundle already carries — a recorded file, a `directory`, or an
-  ancestor directory implied by a recorded file (a bundled `node_modules/dep/index.js`
+- A single-argument `fs.lstatSync(path)`/`fs.statSync(path)` records the path's **kind**
+  — and only its kind. At capture the real call runs first (the program always gets the
+  genuine `Stats`, and genuine errors); the observed type is then stored as a
+  **payload-free** stat record, `stat:file` or `stat:directory` in `formats` — no bytes,
+  no hash, no `files` entry — so a path that is *only* stat'd (never read, listed, or
+  imported) still answers its type getters on load. Only regular files and directories
+  are modelled: a symlink (under `lstat`), socket, or FIFO records nothing, and a
+  `statSync` through an in-root symlink records the *target's* kind, keyed at the
+  requested path exactly like a byte read of it would be. A path already carried as
+  content (read/listed/imported), or whose real location
+  escapes the root, records no stat entry; reading a stat-only path later (same run or a
+  `lock=add`/`bundle=add` re-run) upgrades the weak stat record to the real content
+  record instead of conflicting. On a load
+  run, for a path the bundle carries — a recorded file, a `directory`, a stat record, or
+  an ancestor directory implied by a recorded path (a bundled `node_modules/dep/index.js`
   proves `node_modules` and `node_modules/dep` are directories) —
   `.isFile()`/`.isDirectory()` answer from the bundle so existence checks succeed even
   when the path is absent from disk. Other `Stats` fields/methods are the real stat's
@@ -392,13 +413,16 @@ Captures live in the usual `sources`/`modules` buckets, tagged in `formats`:
   `isFile()`/`isDirectory()` off the result (graceful-fs's `statFixSync` reads
   `uid`/`gid`, for example) keep working under `--bundle=load` instead of re-throwing
   `ENOENT`. Any path the bundle does not carry passes straight through to the real
-  call.
+  call. Stat calls with options (`bigint`, `throwIfNoEntry`) pass through untouched,
+  like `readdirSync` with options.
 - `fs.existsSync(path)`, `fs.accessSync(path)` and `fs.realpathSync(path)` (plus the async
   `fs.access`/`fs.realpath` and `fs.promises.*` forms) are **existence/canonical-path
   probes**, served (load run, for a path the bundle carries) and otherwise passed through.
-  Like `lstat`/`stat` they are **serve-only — never captured**: a probe records nothing, so a
-  file that is *only* existence-probed (never read or imported) is not in the bundle and its
-  probe falls through to disk on load. They matter because build tools check a file *before*
+  Unlike `lstat`/`stat` they are **serve-only — never captured**: a probe records nothing, so a
+  file that is *only* existence-probed (never read, stat'd, or imported) is not in the bundle
+  and its probe falls through to disk on load — though a path the program *does* `lstat`/`stat`
+  gains a stat record, from which these probes then answer too (they consult the same
+  kind lookup). They matter because build tools check a file *before*
   reading it on a channel that never touches its bytes: `@babel/core`, for one, guards config
   loading with `fs.existsSync(babel.config.js)` and realpath-canonicalizes the path **before**
   it `require()`s the file — so without serving the probe, a bundled-but-absent `babel.config.js`
@@ -430,9 +454,11 @@ way. To capture a `.map` instead (e.g. you genuinely ship one as data), add `map
 and once a `.map` is in the bundle it is served on load by membership, even if that run
 omits `--resources=map`.
 
-The two recorded kinds are hashed like any other content (the `directory` integrity
+The two content kinds are hashed like any other content (the `directory` integrity
 is the sha512 of its JSON text), so the lockfile attests them and a frozen run
-verifies them. `--fs` (either mode) requires an active bundle mode (`add`, `replace`, or `load`).
+verifies them; a stat record has no content to hash — the `formats` entry itself is
+the attestation, cross-checked bundle-vs-lockfile like every other format. `--fs`
+(either mode) requires an active bundle mode (`add`, `replace`, or `load`).
 
 When a bundler plugin runs with its own `bundleFile` (a *sidecar*), `--fs` coordinates with
 it so the bundler's module graph isn't duplicated into the main bundle: a read the sidecar

stasis-core/src/fs.js

@@ -2,7 +2,8 @@
 // the filesystem readers
 // fs.readFileSync, fs.readdirSync, fs.lstatSync, fs.statSync so that, alongside the
 // module graph the loader hooks already capture, a program's explicit file/directory
-// reads are recorded into the bundle (bundle=add|replace) and served back from it
+// reads -- and the KIND (file vs directory) of the paths it lstat/stats -- are
+// recorded into the bundle (bundle=add|replace) and served back from it
 // (bundle=load). Alongside the readers, the existence/canonical-path PROBES existsSync /
 // accessSync / realpathSync are served (load mode) so a tool that checks a file before
 // reading it finds a bundle-only file -- see the existsSync/accessSync/realpathSync bullet.
@@ -31,17 +32,27 @@
 //   - readFileSync: the optional second argument (an `encoding` string, or an
 //     options object with `encoding`) is honoured; the raw bytes are what we
 //     capture/serve and `encoding` is applied on the way out.
-//   - lstatSync / statSync: serve-only, bundle=load, single-argument form. For a
-//     recorded path they return a Stats-like object whose isFile()/isDirectory() come
+//   - lstatSync / statSync: single-argument form, captured AND served -- but ONLY the
+//     path's kind, i.e. the type getters isFile()/isDirectory(); full Stats fields are
+//     never attested. CAPTURE (bundle=add|replace): the real call runs first, so the
+//     caller always gets the genuine Stats (and genuine errors); the observed kind is
+//     then recorded as a PAYLOAD-FREE stat record ('stat:file'/'stat:directory' in
+//     `formats` -- no bytes, no hash), so a stat-ONLY path -- never read, readdir'd,
+//     or imported -- still answers at load. A path that is neither a regular file nor
+//     a directory (a symlink under lstat, a socket, a FIFO, ...) isn't modelled and
+//     records nothing. SERVE (bundle=load): for a recorded path they return a
+//     Stats-like object whose isFile()/isDirectory() come
 //     from the bundle (the common existence check); every other member is the real
 //     stat while the file is still on disk, and a benign synthetic default (0 / epoch
 //     / file-or-dir mode) once it's gone -- so fs wrappers that read more than
 //     isFile()/isDirectory() (graceful-fs reads stats.uid/gid) keep working under
-//     bundle=load instead of re-throwing ENOENT. Not patched in capture mode (the file
-//     is on disk there, so the real Stats is correct), except to report a skipped
-//     source-map sidecar as absent (next bullet). statSync vs lstatSync differ only in
-//     symlink following, which matters solely for that on-disk passthrough -- each falls
-//     back to its own real implementation.
+//     bundle=load instead of re-throwing ENOENT. A skipped
+//     source-map sidecar is reported absent in both modes (next bullet). statSync vs
+//     lstatSync differ only in symlink following, which matters for the real-call
+//     paths (passthrough, capture, present-on-disk serve) -- each uses its own real
+//     implementation, and a stat of an in-root symlink records the TARGET's kind
+//     under the requested path (keyed exactly like a byte read of that path would
+//     be) while an lstat of one records nothing.
 //   - existsSync / accessSync / realpathSync (+ their async/promise forms under =async):
 //     existence/canonical-path PROBES, serve-only and bundle=load only. A path the bundle
 //     records answers existent / accessible (read-only: F_OK/R_OK, while W_OK/X_OK defer) /
@@ -57,7 +68,9 @@
 //     module loader). Because these are serve-only (never captured), a file that is ONLY
 //     existence-probed -- never read or imported -- isn't recorded, so its probe falls
 //     through to disk on load; babel.config.js is fine because it is also require()d, hence
-//     recorded. realpathSync returns the real (symlink-resolved) path while the file is on
+//     recorded. (An lstatSync/statSync of a path, by contrast, DOES record its kind -- see
+//     the lstat/stat bullet above -- after which these probes answer from that stat record
+//     too, since they consult the same getFsStatFamily.) realpathSync returns the real (symlink-resolved) path while the file is on
 //     disk, and falls back to the LEXICALLY-resolved request path once it's bundle-only (a
 //     symlink in a surviving ancestor dir is not re-resolved -- moot for the absent-config
 //     case, but a canonicalize-then-compare under a symlinked ancestor could diverge between
@@ -392,18 +405,45 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
     return realReaddirSync(path, options)
   }
 
+  // Capture-side stat recording, shared by the sync/async lstat/stat wrappers: record
+  // the path's KIND (file vs directory) as a payload-free stat record (state.addFsStat),
+  // so the type getters -- lstatSync(x).isFile()/.isDirectory(), the common existence-
+  // and-kind check -- of a stat-ONLY path answer from the bundle at load. Only those two
+  // kinds are modelled (the serve shim's contract); a symlink (under lstat), socket,
+  // FIFO, or device records nothing. The caller has already received (or will receive)
+  // the REAL Stats -- recording never alters what the program sees. Guards mirror the
+  // byte readers: a path whose real location escapes the root (in-tree symlink pointing
+  // out) is not attested, a file whose bytes a write-mode sidecar already carries is
+  // skipped, and a recording conflict (a kind flip mid-run) taints the capture via
+  // markAborted so nothing inconsistent is written.
+  const captureStat = (state, abs, stats) => {
+    const isDir = stats.isDirectory()
+    if (!isDir && !stats.isFile()) return
+    const url = pathToFileURL(abs).toString()
+    if (!realContained(state.root, abs) || state.attestedBySidecar(url)) return
+    try { state.addFsStat(url, isDir ? 'directory' : 'file') } catch (err) { markAborted(err) }
+  }
+
   fs.lstatSync = function lstatSync(path, options) {
     const state = getState()
-    // Serve-only, single-argument form. Capture mode is left to the real lstatSync
-    // (the file is on disk, so its Stats is correct and complete) -- the lone exception
-    // being a skipped source-map sidecar, reported absent in both modes.
+    // Single-argument form. Serve (bundle=load) the bundle-backed type getters for a
+    // recorded path; capture (bundle=add|replace) the observed kind via captureStat,
+    // handing the caller the REAL Stats (and real errors, e.g. ENOENT) untouched.
+    // A skipped source-map sidecar is reported absent in both modes.
     if (state && options == null) {
       const abs = toAbsPath(path)
       if (abs !== null && withinRoot(state.root, abs)) {
         if (isSkippedSourceMap(state, abs)) throw enoent('lstat', path)
         if (state.config.loadBundle) {
           const kind = state.getFsStatFamily(pathToFileURL(abs).toString())
           if (kind !== undefined) return bundleStats(realLstatSync, path, kind === 'directory')
+        } else if (state.config.writeBundle && !isLoadingModule()) {
+          // (Node's loader stats module paths through internal bindings, not this
+          // public API, but skip during a load window anyway, symmetric with
+          // readFileSync/readdirSync.)
+          const stats = realLstatSync(path)
+          captureStat(state, abs, stats)
+          return stats
         }
       }
     }
@@ -413,7 +453,10 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
   // statSync mirrors lstatSync exactly; the only difference is symlink following,
   // which is moot for a bundle-served path (there is no link to follow once the
   // bytes come from the bundle) but matters for the passthrough/present-on-disk
-  // case, so it falls back to the real statSync rather than lstatSync.
+  // case AND for capture -- a statSync of an in-root symlink records the TARGET's
+  // kind (keyed at the requested path, like a byte read of it), while an lstatSync
+  // of it records nothing (a symlink isn't modelled) -- so each uses its own real
+  // implementation.
   fs.statSync = function statSync(path, options) {
     const state = getState()
     if (state && options == null) {
@@ -423,6 +466,10 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
         if (state.config.loadBundle) {
           const kind = state.getFsStatFamily(pathToFileURL(abs).toString())
           if (kind !== undefined) return bundleStats(realStatSync, path, kind === 'directory')
+        } else if (state.config.writeBundle && !isLoadingModule()) {
+          const stats = realStatSync(path)
+          captureStat(state, abs, stats)
+          return stats
         }
       }
     }
@@ -597,8 +644,10 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
       return realReaddirP(path, options)
     }
 
-    // lstat/stat are serve-only (like their sync siblings); capture is left to the real
-    // call. bundleStats's field fallback is synchronous, so it uses the sync real stat.
+    // lstat/stat capture and serve exactly like their sync siblings: serve the
+    // bundle-backed type getters, or record the real call's observed kind via
+    // captureStat while handing the caller the real Stats. bundleStats's field
+    // fallback is synchronous, so it uses the sync real stat.
     fs.lstat = function lstat(path, options, callback) {
       const cb = typeof options === 'function' ? options : callback
       // "single-argument" form: fn(path, cb) OR fn(path, null/undefined, cb). graceful-fs
@@ -610,6 +659,13 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
       if (t?.mode === 'serve') {
         const kind = t.state.getFsStatFamily(t.url)
         if (kind !== undefined) { queueMicrotask(() => cb(null, bundleStats(realLstatSync, path, kind === 'directory'))); return }
+      } else if (t?.mode === 'capture') {
+        realLstat(path, (err, stats) => {
+          if (err) return cb(err) // faithful error on the user's original path; nothing recorded
+          captureStat(t.state, t.abs, stats)
+          cb(null, stats)
+        })
+        return
       }
       return realLstat(path, options, callback)
     }
@@ -625,6 +681,13 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
       if (t?.mode === 'serve') {
         const kind = t.state.getFsStatFamily(t.url)
         if (kind !== undefined) { queueMicrotask(() => cb(null, bundleStats(realStatSync, path, kind === 'directory'))); return }
+      } else if (t?.mode === 'capture') {
+        realStat(path, (err, stats) => {
+          if (err) return cb(err)
+          captureStat(t.state, t.abs, stats)
+          cb(null, stats)
+        })
+        return
       }
       return realStat(path, options, callback)
     }
@@ -635,6 +698,10 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
       if (t?.mode === 'serve') {
         const kind = t.state.getFsStatFamily(t.url)
         if (kind !== undefined) return bundleStats(realLstatSync, path, kind === 'directory')
+      } else if (t?.mode === 'capture') {
+        const stats = await realLstatP(path)
+        captureStat(t.state, t.abs, stats)
+        return stats
       }
       return realLstatP(path, options)
     }
@@ -645,6 +712,10 @@ export function installFsHooks({ async: patchAsync, getState, markAborted, isLoa
       if (t?.mode === 'serve') {
         const kind = t.state.getFsStatFamily(t.url)
         if (kind !== undefined) return bundleStats(realStatSync, path, kind === 'directory')
+      } else if (t?.mode === 'capture') {
+        const stats = await realStatP(path)
+        captureStat(t.state, t.abs, stats)
+        return stats
       }
       return realStatP(path, options)
     }