executeCode: every binding call fails with "Subrequest depth limit exceeded" on production since #177

[pol-cc]

Since #177, on a real Cloudflare production deployment, the first outbound call from executeCode to any binding fails:

Error: Subrequest depth limit exceeded. This request recursed through Workers too many times.
    at async default (agent.js:2:19)

This is not specific to one gatekeeper: it hits gatekeeper bindings and gadget RPC (env.<GADGET>.<method>()) alike, which makes executeCode unusable for anything that touches the outside world. Intermittently the same calls surface as Timed out waiting for logs from code execution instead, which I suspect is the same failure arriving through a path that can't report it.

Repro

The entire executeCode block — nothing else in it:

export default async function(self, env, ctx) {
  const actives = await env.SCHEDULER.list();
  console.log(JSON.stringify(actives, null, 1));
}

console.log from the same invocation runs fine when placed before the call, so the module loads and starts up correctly. What dies is the first outbound call.

Regression window

Works at 213ea6a. Broken at dd2b015. Same workspace, same chat, same code, ~1 hour apart; the only thing that changed in between was the deployed backend. Before the upgrade the same session had made dozens of binding calls (BigQuery reads through a gatekeeper, gadget RPC, SCHEDULER.calendarAt) without a single incident.

Root cause

#177 changed how the code-mode worker is loaded in overseer.ts:

-      let restoreGadgetId = this.executeCodeRestoreTarget();
-      if (restoreGadgetId === undefined) {
-        entrypoint = this.env.LOADER.load(workerDef).getEntrypoint<CodeModeEntrypoint>();
-      } else {
-        // Wacky hack: Load the code mode dynamic worker through `ctx.restore()` [...]
-        let codeId = crypto.randomUUID();
-        try {
-          this.#codeIdMap.set(codeId, workerDef);
-          entrypoint = await this.ctx.restore({
-            type: "gadget",
-            gadgetId: restoreGadgetId,
-            codeId,
-          });
-        } finally {
-          this.#codeIdMap.delete(codeId);
-        }
-      }
+      let entrypoint = this.env.LOADER.load(workerDef).getEntrypoint<CodeModeEntrypoint>();

Loading the worker directly with LOADER.load() from the in-flight request appears to leave it one level deeper in the subrequest chain than the ctx.restore() path did. A binding call from executed code already traverses several Workers — executeCode worker → GatekeeperLoopback → Overseer DO → gatekeeper Worker → and back into the Overseer for authorizeObservation() — so one extra level is enough to cross the limit on the first call.

I can't verify the runtime semantics of ctx.restore() vs LOADER.load() from outside, so treat that last paragraph as the mechanism I inferred rather than something I confirmed. The causality itself is solid, though — see below.

Why this wouldn't show up in CI or local dev

The "recursed through Workers too many times" cap is enforced by the production platform. wrangler dev / local workerd don't apply it the same way, so a deployment to real Cloudflare is the first place it can bite.

Confirmed by the workaround

We patched our deployment to restore the pre-#177 load path — keeping the direct LOADER.load() for the no-gadget case exactly as before — and changed nothing else. The same minimal block above went straight from failing to returning [], and binding calls work normally again. Same environment, same call, single-variable change.

The forging added by #177 is unaffected by that revert: env.<GADGET>[restore]() goes through forgeRestoreStubForBinding(), which loads its own worker under its own codeId.

I have that as a ~20-line patch and I'm happy to open it as a PR, but I'd rather not presume — you removed that ctx.restore() deliberately, and you may prefer to keep LOADER.load() and cut depth somewhere else in the chain. Just say which you'd prefer.

Related, minor

With several env.<GADGET>[restore]() calls in a single executeCode invocation, #soleForgedRestoreTarget() returns undefined (because targets.size > 1) and bindHook() attributes every resulting hook to the workspace's first gadget. The callbacks themselves target the right gadgets — your comment says as much ("for bookkeeping; the callback itself already encapsulates the correct restore target") — but since the Connections tab filters hooks by gadget, the hooks become unreachable from the dashboard they belong to, and there's no error to hint at it. Registering one per invocation works fine. This is the TODO already sitting next to that code; happy to file it separately if you'd like it tracked.

Deployment context

Self-hosted production deployment on a customer Cloudflare account: 8 Workers (Workshop + 7 gatekeepers, some of them our own) wired through service bindings, Workshop behind Cloudflare Access, no AI Gateway. Glad to provide more detail or test a fix.