[REA-5060] Add distributed worker protocol and frame transport - #134
Open
Orion-Zheng wants to merge 7 commits into
Open
Orion-Zheng wants to merge 7 commits into
Orion-Zheng wants to merge 7 commits into
Conversation
The standalone runtime has no multi-GPU story: a model that needs several GPUs has nowhere to put the per-device half of itself, so the abstraction that solved this in the previous runtime is brought across. This commit is only the move. The six files land byte-identical, so this commit is the baseline every later commit in the stack is reviewed against — it proves nothing was lost in transit, and it separates "the code arrived" from "the code was adapted". The adaptations this repo requires (its docstring style, its typing strictness, no references to systems an outside reader cannot see) all follow as their own commits on top, where they can be read as deliberate changes rather than hidden inside a 750-line paste. Nothing imports the package yet and nothing re-exports it, so this commit does not change behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Andy <andy@reactor.inc>
The move landed the code unchanged; this makes it belong here. Four kinds of change, none of them behavioural — the ten protocol tests that came with the package pass untouched, which is what makes that claim checkable. Style and provenance. The copyright headers go, since this repo carries none. References to systems an outside reader cannot see go too, with their content kept where it was load-bearing: "a standing watchdog is not implemented" says everything a reader needs without naming a tracker, and the dangling pointer to an example that was never published is simply dropped. Docstrings gain the one-line imperative summary the linter wants; the Sphinx roles stay, because this repo already mixes them with Google sections. Logging. This is the one seam that would have failed at runtime. get_logger has the same name in both runtimes but returns a different object here: a structured logger whose level methods take fields as keywords and reject positional args. Two calls passed them, and both sit on failure paths — the worst place to learn that a logging call raises. Every call now uses a static message plus fields, and a spawned worker installs the runtime's own formatter instead of a bare one, so its lines share the runtime's shape and carry the rank as a field rather than a string prefix. Surface. The five public names are re-exported from the top-level package, because the supported surface here is what that package exports; a name reachable only through a submodule reads as unsupported. What makes this safe is that torch is imported lazily, inside the functions that use it, so importing the runtime still pulls in nothing heavier than numpy. That rests on discipline rather than on structure, so a test now pins it: a subprocess imports the package and asserts torch never entered sys.modules. Typing. Only two things needed saying. torch is an optional dependency the checker cannot see, so its lazy imports carry a narrow suppression with the reason attached. And a concrete setup() override necessarily narrows the base's **setup_kwargs, which reads as an incompatible override — a property of the abstraction that every model's worker will meet, so the base class now documents it and the test suppresses it the same way a model would. Signed-off-by: Andy <andy@reactor.inc> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Andy <andy@reactor.inc>
The README is the front page, and it described a runtime where a model runs on one device. Multi-GPU changes what an author can build with it, so it belongs here rather than only in the API docs. One line in Highlights, and a paragraph in How it works placed after the single-GPU example — because the point worth making is that the example does not change. The model keeps its event loop, its session, and its output stream, and gains a handle it drives from the same run() loop. Framed that way rather than as a feature list, since the reason to reach for this is a model that outgrew one device, not a wish to write distributed code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Andy <andy@reactor.inc>
Orion-Zheng
changed the title
Orion-Zheng
marked this pull request as ready for review
August 14, 2026 16:55
Contributor
|
The engine is a good approach. Failure taxonomy, the max-end-row trick, the shared-memory path, liveness, the teardown ladder, the ten protocol tests are solid. The changes I'd like to see are the author surface. We need to be careful as once the names are importable they are minimally an implicit contract and likely to be taken as an explicit contract. Today multi-GPU is a second way to write a model ( Distributed by default, single-GPU is world_size=1:
The base class is the paved road that covers essentially every model. The raw primitives stay available as experimental internals for the rare model that genuinely needs to hand-drive. That demotes them from "the API" to "the escape hatch" instead of removing the capability. Think about DX ergonomics when deciding to either keep The README documents This is moving the driving into a base class and thinning the surface. If my recommendations are wildly out of left field or otherwise would pose too many ergonomic challenges, please let me know. |
Review follow-up (#134): the distributed primitives are the raw layer a future runtime-managed path will drive, so they should not harden into the stable top-level API before that path exists. - Drop the five distributed re-exports from the reactor_runtime root; they are imported from reactor_runtime.distributed only, and the package test now asserts they stay off the root rather than on it. - Mark the surface experimental in the package docstring and README, and state the direction: the runtime owns the group wiring, the primitives remain the escape hatch for layouts the managed path cannot express. - Recommend sourcing world_size from model config instead of a literal in load(). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Kavya <kavya@reactor.inc>
Provide an experimental video adapter with managed worker sessions, control snapshots, pause/reset, and cleanup. Derive worker count from the manifest, keep one-worker execution in-process, and include a runnable example with actionable errors. Include current main while preserving the original PR commits. Signed-off-by: Kavya <58096384+maxmb1@users.noreply.github.com>
This stack of pull requests is managed by Graphite. Learn more about stacking. |
## Why Multi-GPU models need per-rank lifecycle hooks and bounded frame transport without committing the package root to a stable distributed API. This is the foundation of the REA-5060 stack and can land independently of an engine host. ## What Changed Adds the experimental single-node DistributedWorker protocol, worker error types, and shared uint8 video frame transport. Torch remains lazy and optional at import time, and package tests keep the surface off the root exports. Frame-buffer and package tests accompany this layer; worker-group orchestration, the managed model adapter, and the runnable example are separate dependent PRs. Local Python 3.12 unit/contract tests, lint, and type checking pass. No NVIDIA/NCCL execution is claimed. Signed-off-by: Kavya <58096384+maxmb1@users.noreply.github.com>
maxmb1
changed the title
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Multi-GPU models need per-rank lifecycle hooks and bounded frame transport without committing the package root to a stable distributed API. This is the foundation of the REA-5060 stack and can land independently of an engine host.
What Changed
Adds the experimental single-node DistributedWorker protocol, worker error types, and shared uint8 video frame transport. Torch remains lazy and optional at import time, and package tests keep the surface off the root exports. Frame-buffer and package tests accompany this layer; worker-group orchestration, the managed model adapter, and the runnable example are separate dependent PRs. Local Python 3.12 unit/contract tests, lint, and type checking pass. No NVIDIA/NCCL execution is claimed.