myst build --execute reports a dead kernel as a bare 'Unhandled error' — no notebook, cell, cause, or kernel stderr

[mmcky]

Summary

When a notebook's kernel dies during myst build --html --execute, the build fails with a bare Unhandled error and the CLI version banner — it never says which notebook, which cell, why the kernel died, or that a kernel death is what happened. Diagnosing one of these in CI took instrumented runs with an out-of-band memory sampler and a dmesg read-back; the information myst had in hand at failure time would have shortened that to one look at the log.

What the log shows

From QuantEcon/lecture-python-programming run 30806359596 (the instrumented reproduction; runs 30801749639 and 30802052003 are identical uninstrumented failures), myst build --html --execute on a 25-notebook site, fork at qe-v8:

10:41:59  💿 Executing notebook (jax_intro.md) [no execution cache found]
10:43:05  Connection lost, reconnecting in 0 seconds.
10:43:05  Kernel: restarting (df32cb96-...)
10:43:05  Kernel: autorestarting (df32cb96-...)
10:43:05  Starting WebSocket: ws://localhost:35657/api/kernels/df32cb96-...
10:43:05  Unhandled error

Myst CLI Versions:
 - node   24.18.1
 - npm    11.16.0
 - myst   1.10.1
##[error]Process completed with exit code 1

The kernel id is printed but the notebook is not, and on a build executing many notebooks concurrently the mapping is not recoverable from the log — the failing notebook had to be inferred by diffing Executing notebook (...) lines against Built ... lines. The actual cause (the kernel process aborted after a GPU memory-pool exhaustion — an XLA fatal in the child process, nothing to do with myst) was only established with a 5-second GPU/RSS sampler running beside the build and a dmesg check to rule out the host OOM killer.

Separately, the kernel had been dead for roughly 40 seconds before the build noticed at 10:43:05 — the death itself happened around 10:42:25 by the GPU telemetry. Not a bug, but it means the timestamp in the log points well away from the event being debugged.

What would have made this a one-look diagnosis

In rough order of value:

1. Name the notebook (and cell index if known) in the failure path. The executor knows which notebook owns kernel df32cb96 and which cell was in flight. Kernel died while executing jax_intro.md (cell 41) instead of Unhandled error.
2. Say that a kernel death is what happened. autorestarting followed by an unhandled rejection is the internal symptom; the user-facing fact is "the kernel process exited unexpectedly". Reporting it as such (with the exit code/signal if jupyter-server exposes it) turns an opaque crash into a directed hint — a signal points at the OS, a clean nonzero exit points in-process.
3. Surface the kernel's stderr tail. An XLA fatal writes its reason to the kernel's stderr before aborting. Capturing the last ~50 lines per kernel and printing them for a dead kernel would very likely have named the allocation failure directly.
4. Fail the page, not the process, where possible. A single kernel death currently surfaces as an unhandled rejection that takes down the whole build after the autorestart. Failing that page's execution with a proper error (and letting --keep-going-style semantics decide whether the build continues) would match how the rest of the build treats per-page errors.

Context

This is very likely inherited from upstream myst-cli's execution path rather than fork-specific — filing here first because this CLI is what the QuantEcon lecture builds run, and it may be worth carrying upstream with the rest of the UPSTREAM-PRS.yml queue. Full incident write-up: QuantEcon/lecture-python-programming#363 (comments from 2026-08-03), root cause and fix in commits 94a7e52 and f541ccc there.