feat(server): keep pool sandbox sessions on their supervisor's gateway by FrostGod · Pull Request #5052 · NVIDIA/OpenShell

FrostGod · 2026-10-29T23:57:48Z

Summary

Each pool sandbox session dials the gateway replica that serves its supervisor, so a sandbox and its parent supervisor always share a gateway.

Closes #5051

Changes

HostAccepted carries the accepting replica's peer endpoint.
The supervisor host publishes it; its sandbox instances dial their sessions there (redirected=true, so the replica serves them locally). They wait up to 5 s for it after a host reconnect.
A stopping gateway refuses new host streams and ends existing ones, so hosts reattach to a live replica and their sessions follow.
After a host stream ends, for example on supervisor token expiry, the host redials the replica that accepted it before falling back to the Service, so its sessions stay put.
An established session reconnects only if its host does land on another replica.
e2e:kubernetes:shared-supervisor-ha now runs two gateway replicas on PostgreSQL and checks colocation through gateway and supervisor failures.

flowchart LR
  G1[gateway A] -- host stream --> S1[supervisor 1]
  S1 -. sessions dial A .-> G1
  G2[gateway B] -- host stream --> S2[supervisor 2]
  S2 -. sessions dial B .-> G2

Testing

Unit tests: openshell-supervisor-control, openshell-supervisor-process, openshell-supervisor, openshell-server
The colocation test on k3d (2 gateways, PostgreSQL, 2 supervisors, 4 sandboxes): sessions stay on their supervisor's gateway and exec works after start, a gateway crash, a gateway rolling restart, and a supervisor crash, with 60 s supervisor tokens so hosts also move between gateways on their own

copy-pr-bot · 2026-10-29T23:57:59Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…lica A shared supervisor pool sandbox's session now goes to the gateway replica serving its owning member's stream, instead of the replica the ring picks by sandbox ID. Each pool reconciliation pass publishes the sandbox-to-replica map; sessions fall back to the ring when the parent is unknown, not a live peer, or this replica is shutting down. Signed-off-by: divesh <dgude@nvidia.com>

A pool sandbox session served by a replica other than the one serving its owning member is redirected there on its next heartbeat, reusing the shutdown handoff so the sandbox stays Ready. This re-homes sessions left on the wrong replica by a rolling restart. Signed-off-by: divesh <dgude@nvidia.com>

Removes the per-sandbox parent map and the heartbeat relocation; pool sessions will dial their supervisor's gateway directly instead. Signed-off-by: divesh <dgude@nvidia.com>

…plica HostAccepted now carries the accepting replica's peer endpoint. A shared host's sandbox sessions dial it directly, so each session runs on the replica serving its supervisor. A stopping replica sheds its host streams, and redirected sessions follow the host to its next replica. Closes #5051. Signed-off-by: divesh <dgude@nvidia.com>

…ailover Add the shared-supervisor-tree lane: two gateway replicas on PostgreSQL with a two-member supervisor pool. After start, a gateway crash, a gateway rolling restart, and a supervisor crash, every pool sandbox's session runs on the replica serving its supervisor and exec reaches every sandbox. Signed-off-by: divesh <dgude@nvidia.com>

github-actions · 2026-10-30T22:14:18Z

🌿 Preview your docs: https://nvidia-preview-pr-5052.docs.buildwithfern.com/openshell

drew · 2026-10-30T23:02:43Z

Established sandbox sessions do not follow a host that reconnects elsewhere. Before this PR, session placement followed the hash ring independently of the host. This PR colocates sessions initially, but reads gateway_home only between connection attempts. Once run_single_session(...).await establishes a healthy session, later home changes are never observed.

This can happen during normal supervisor-token expiry: the gateway closes the host's assignment stream, and the host reconnects through the Service, potentially landing on another replica. Existing sandbox sessions remain on the previous gateway indefinitely, breaking the promised tree topology.

Please watch for home changes while a session is active and reconnect when a different nonempty endpoint arrives. Add coverage that moves only the host stream while keeping sandbox sessions healthy; the new gateway crash/restart tests close both and miss this case.

Validation: reviewed the full diff and surrounding lifecycle code; did not run tests locally.

drew · 2026-10-30T23:07:53Z

Can we fold this coverage into e2e:kubernetes:shared-supervisor-ha instead of adding a new lane? We could run that lane with two gateways and PostgreSQL, add the colocation assertions and gateway failure/restart steps, and keep its existing checks that workloads survive without restarting.

…es gateway A host whose stream ends, for example when its supervisor token expires, can reattach to another replica. Its sandbox sessions now watch for that and reconnect to the host's new replica instead of staying on the old one. Signed-off-by: divesh <dgude@nvidia.com>

…-ha lane Run the shared-supervisor-ha lane with two gateway replicas on PostgreSQL and move the gateway crash, rolling restart, and supervisor crash colocation test into it, replacing the separate shared-supervisor-tree lane. Signed-off-by: divesh <dgude@nvidia.com>

…ream ends A host's stream ends routinely, for example when its supervisor token expires. Redialing the replica that accepted it, before falling back to the Service, keeps the host and its sandbox sessions on that replica instead of moving them all on every reconnect. Signed-off-by: divesh <dgude@nvidia.com>

FrostGod · 2026-11-01T19:14:42Z

Established sandbox sessions do not follow a host that reconnects elsewhere. Before this PR, session placement followed the hash ring independently of the host. This PR colocates sessions initially, but reads gateway_home only between connection attempts. Once run_single_session(...).await establishes a healthy session, later home changes are never observed.

This can happen during normal supervisor-token expiry: the gateway closes the host's assignment stream, and the host reconnects through the Service, potentially landing on another replica. Existing sandbox sessions remain on the previous gateway indefinitely, breaking the promised tree topology.

Please watch for home changes while a session is active and reconnect when a different nonempty endpoint arrives. Add coverage that moves only the host stream while keeping sandbox sessions healthy; the new gateway crash/restart tests close both and miss this case.

Validation: reviewed the full diff and surrounding lifecycle code; did not run tests locally.

Good find, this is actually much bigger than I thought. I didn't know the host stream gets closed on every token expiry, so hosts were hopping gateways and leaving their sessions behind. Fixed it two ways: the host now redials the same gateway first, so nothing moves in the normal case, and sessions follow the host if it does end up somewhere else. Tested on k3d with short tokens, hosts stayed put across expiries.

FrostGod · 2026-11-01T19:15:11Z

Can we fold this coverage into e2e:kubernetes:shared-supervisor-ha instead of adding a new lane? We could run that lane with two gateways and PostgreSQL, add the colocation assertions and gateway failure/restart steps, and keep its existing checks that workloads survive without restarting.

@drew done.

FrostGod requested a review from drew October 30, 2026 00:05

FrostGod added 5 commits October 30, 2026 14:38

refactor(server): drop gateway-side pool session placement

465aa71

Removes the per-sandbox parent map and the heartbeat relocation; pool sessions will dial their supervisor's gateway directly instead. Signed-off-by: divesh <dgude@nvidia.com>

FrostGod force-pushed the feat/5051-supervisor-tree-routing/FrostGod branch from 68a5a70 to efda3de Compare October 30, 2026 22:13

FrostGod added 3 commits October 31, 2026 18:12

Navigation Menu

Repository navigation

feat(server): keep pool sandbox sessions on their supervisor's gateway - #5052

feat(server): keep pool sandbox sessions on their supervisor's gateway#5052
FrostGod wants to merge 8 commits into
mainfrom
feat/5051-supervisor-tree-routing/FrostGod

FrostGod commented Oct 29, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

drew commented Oct 30, 2026

Uh oh!

drew commented Oct 30, 2026

Uh oh!

FrostGod commented Nov 1, 2026

Uh oh!

FrostGod commented Nov 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FrostGod commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Uh oh!

copy-pr-bot Bot commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 30, 2026

Uh oh!

drew commented Oct 30, 2026

Uh oh!

drew commented Oct 30, 2026

Uh oh!

FrostGod commented Nov 1, 2026

Uh oh!

FrostGod commented Nov 1, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FrostGod commented Oct 29, 2026 •

edited

Loading