tokenmaxxer: the meeting room that ships while you talk#1
Merged
Conversation
- Gateway bridges OpenAI Realtime-compatible clients to Inworld upstream - Server owns session config; clients only choose turn detection - Tool calls executed server-side, dual wire-shape, deduped by call_id - Malformed (nameless) tool calls dropped + dead-air recovery nudge - check_progress schema requires a param: empty-properties schemas make gemma-4 emit nameless function calls - jimmy speed chain outputs JSX body only, file skeleton applied in code; gemini thinking disabled; overall codegen deadline - Playground file-per-page routing with crash boundary; console with mic, playback worklets, transcript, tool feed, fleet panel, preview
- STT shim: Inworld doesn't transcribe committed audio under manual turn detection (turn_detection null), so the gateway whispers the buffered turn and injects it upstream as text — scenario voice tests now hear real words end to end - Capacity resilience: transient 'model at capacity' upstream errors are retried, repeated ones switch the session to a fallback model (google-ai-studio/gemini-2.5-flash) mid-session - open_page teaches the model when a page doesn't exist; write_page protects the /home index; Max confirms page success as fact - Voice scenario tests: dispatch-on-idea, progress check, spoken page request — all real audio, real tools, real fleet
- Voice tests persist full.wav + per-turn segments + manifest; agent audio independently verified by whisper transcription - dispatch() registers a provisional fleet entry instantly so the room screen reacts the moment words are spoken, brain confirms or renames - Brain default switched to claude-haiku-4-5: routing decisions 2-4s (was 5-10s), quality verified by the brain tests
- Upstream sendRaw queues until session ready (the console greet raced the Inworld connection and was silently dropped); the handshake session.update bypasses the queue (deadlock otherwise) - Console greets only after session.created - Persona: one request, one tool call - Console grid: arbitrary fr tracks need minmax(0,...) — a long unbreakable tool-arg string blew the layout to 2382px - qa-console script drives the real console with a synthesized wav as the microphone: greeting, VAD turns, tool feed, page on room screen
- experiment-codegen.ts: 8 page briefs x 4 models logged to LangWatch; jimmy p50 525ms vs 9-21s for the rest at 0.66 vs 0.92 quality — chain order is now data, not vibes (numbers in the speedChain comment) - Tracing: brain routing, speed-chain codegen and tool execution spans via langwatch setupObservability; verified arriving with trace search - Gateway auto-starts the jimmy proxy when it's down - DEMO.md: demo arc, pre-flight checks, failure playbook, reset steps - Full suite re-run: 19/19
- findByTopic and the brain's reuse candidates exclude status=gone agents (QA caught a dispatch routed as follow-up to a killed tmux) - semantic VAD eagerness low: medium split one spoken request into 2-3 turns, each re-triggering tools and polluting the model context - write_page output steers the model away from repeat page tools - Multi-turn edit voice scenario (write then edit the same page) - QA script now drives the full demo arc: greet, page, dispatch, fleet panel, progress answer
The Iron Man delta. tokenmaxxer now drives the real desktop, not an iframe: - DesktopController interface wraps all OS specifics; MacDesktopController (osascript) implemented, NoopDesktopController for tests/CI, Windows stub documented for the incoming teammate. Factory picks by platform + the TOKENMAXXER_DESKTOP flag; pure tiler (regionForSlot) is cross-platform. - write_page/edit_page/open_page pop a chromeless Chrome app-window tiled top-right showing the live page (positions itself on the current Space via geometry flags; Vite HMR makes edits appear with no reload). - dispatch_work opens a Warp window running tmux attached to the agent, so the room watches Claude work live; terminals fan out across quadrants. - Proactive: on STATUS.md DONE the fleet fires a macOS notification and pops the PR window, even in silence. - Tests: tiler + noop unit tests (headless), gated macOS integration test that really opens/tracks/closes a window. Existing suites stay headless.
- Page-codegen refactored into a model registry (jimmy, gemini-flash, haiku, inworld-gemma, gpt-4.1-mini) behind a uniform generate(); haiku via Anthropic, inworld-gemma via realtime text session - settings.pageModel is runtime-mutable; set_page_model voice tool switches it, broadcast as tokenmaxxer.settings so the console reflects it; chain = selected primary + smart fallbacks - Benchmark now spans all five models. Fresh numbers: jimmy 0.8s/0.65, inworld-gemma 8s/0.92, gemini-flash 8.6s/0.92, haiku 13s/0.93 (sharpest), gpt-4.1-mini 16s/0.90 - Tests: switching + every available model writes valid TSX (jimmy 4.5s, gemini 6.4s, haiku 9.8s, inworld-gemma 6.5s)
- The tool Max picks is the route: pages → write_page/edit_page (fast model), real work → dispatch_work (Claude agent in a terminal). Persona and the dispatch_work schema now draw the line explicitly (look-at vs runs/reads). - routing.test.ts drives the real gateway with 6 labeled requests; pages route to a page tool, login/research/Stripe route to dispatch_work, all under ~1.3s. - fleetDryRun flag lets routing tests assert tool choice without spawning real tmux agents.
The OpenAI Realtime adapter parked the agent transcript on lastAgentTranscript but returned an audio-only message, so the LangWatch Simulations UI showed blank audio players for Max's turns (the user simulator already attached its transcript, which is why only Max looked empty). maxAgent subclasses the adapter to attach the transcript as a leading text part. A regression assertion proves the agent audio message now carries text. (Same fix belongs upstream in the scenario framework.)
The listening layer split from the prototyping layer. The prototyping is now real desktop windows; the HUD is a frameless, transparent, always-on-top Electron overlay that floats above everything (incl. full-screen apps, all Spaces) showing listening state, the last turn, the last action and the fleet size. - console gains a compact Hud view (?hud=1) reusing the same room client (mic/playback/WS) as the full dashboard — verified floating over other apps with the listen control and live wiring. - hud/ Electron shell wraps it; auto-grants the mic; cross-platform shell (same overlay runs on Windows once the desktop adapter lands there). - pnpm hud launches it; README reframed around desktop control.
- DEMO.md: pnpm hud, accessibility/mic grants, the desktop arc (page window, Warp agent terminal, proactive PR, model switch), new failure modes and a fuller reset - README architecture reframed around desktop control (done in prior commit), HUD run instructions - preflight checks Chrome + a terminal are installed - root: pnpm hud script
The frameless HUD had no way to close it and no guard against stacking, so running pnpm hud twice left multiple invisible-chrome windows on top of each other with no affordance to dismiss them. - ✕ button on the bar (window.close), styled subtle, no-drag so it clicks - Cmd/Ctrl+Q and Cmd/Ctrl+W quit, since a frameless overlay has no menu bar - window-all-closed now quits on every platform: the overlay is the whole app - requestSingleInstanceLock: a second launch focuses the existing window instead of opening another one
The product pivoted: drop chatjimmy/page generation entirely; the core is now the voice agent spinning swarms of Claude Code agents into one shared kanban channel where they self-organize (the channel is the agent loop). - room-orchestration.feature: canonical room behavior (spawn/grow/message a room, the channel-as-ralph-loop, ten-agent tokenmaxxing, open a URL, fix a github issue with a fresh room) - meeting-agent.feature: Max now delegates to rooms + open_url, no page tools - orchestrator.feature: the room engine (one channel + N agents, deterministic naming, no LLM router, project->workspace, reuse by topic) - desktop-control.feature: open_url browser window, KanbanCode app pinned top-left on the room channel, terminals fan out, HUD on top - remove obsolete jimmy-era specs (playground, model-selection, task-routing)
The room no longer renders pages with a fast model; it turns talk into a SWARM. A spoken mission becomes N Claude Code agents that all land in one shared kanban channel and self-organize there. The channel is the agent loop: agents claim slices, review each other, and push the mission forward without Max micromanaging. One can step up as supervisor. Engine (server/src/orchestrator/room.ts): - spawn_room(mission, topic, agents?, project?): creates a channel, launches agents (names from a pool, no tokens spent generating them), gives each a coordinate-first brief (join the channel, read history, claim a slice, then build), pops a terminal per agent, and brings the KanbanCode board forward on the channel. - message_room / add_agents: steer or grow a running room by topic. - check_progress: status drawn from channel history, never invented. - Deterministic project registry resolves a spoken hint (the website, langwatch) to a real checkout + URL, so dispatch is instant with no LLM in the hot path. Tools (Inworld): spawn_room, message_room, add_agents, open_url, check_progress. open_url trusts a real remote URL (a GitHub issue) but maps localhost guesses and bare phrases (the website) to the running site. Desktop: DesktopController gains focusChannel() — deep-links the board to a channel and pins it top-left; terminals fan out, the browser sits top-right. Removed entirely: the jimmy proxy, page codegen, model switching, the LLM routing brain, and their tests. Network-free room.unit tests cover project resolution, channel naming, and the coordinate-first brief; CI runs them. Co-authored-by: tokenmaxxer-alexander <agent@tokenmaxxer>
The unit test imported the engine through room.ts/handlers.ts, which pull in config.ts — and config requires live API keys at import, so CI (no .env) threw 'INWORLD_API_KEY missing' before a single assertion ran. Move the pure helpers the test needs into config-free modules: channelName + buildAgentBrief into orchestrator/brief.ts, and the open_url resolver into tools/url.ts. The test now imports only those + the project registry, and passes with no .env. Also rewrites the README around the room-swarm model.
Preflight drops the jimmy proxy and playground checks, adds a KanbanCode running check and a kanban-channel round-trip (create + send + history) that proves the agent-loop mechanism, and swaps the playground for the live site. DEMO.md rewrites the arc around spinning up rooms of self-organizing agents, steering them by voice, and running two missions in parallel.
The delegation eval (scripts/experiment-delegation.ts) showed open_url over-firing as gemma-4's catch-all whenever it was unsure: 'throw two more agents at it', 'how's it going in there', and bare 'redesign the X' all misrouted to open_url. Tighten the tool descriptions and the persona so each move claims its own trigger words and explicitly disclaims open_url, and spell the routing out: SHOW -> open_url, DO -> spawn_room, STEER -> message_room/add_agents, STATUS -> check_progress. Also short-circuit launchAgent under fleetDryRun so add_agents into a live room is exercisable headlessly (it reaches launchAgent directly, bypassing launchRoom's guard).
- routing.test.ts: delegation routing (spawn_room / open_url / message_room / add_agents / check_progress) replacing the old page-vs-work routing - gateway.test.ts: open_url + 'No rooms running' in place of write_page and the Fleet-empty assertion - voice/max.voice.test.ts: room scenarios (spin up a room, open a url, status, open-issue-then-spin-a-fix-room); resets state on start - scripts/experiment-delegation.ts: LangWatch delegation eval — labeled utterances scored against the tool Max picks, context-aware for follow-ups - tests/helpers/reset.ts + scripts/reset-rooms.ts: one-command clean slate (kill room agents, delete room channels, baseline the served site)
The earlier accuracy (47-59%) understated reality: spawning a room in the same conversation right before a steer/status case biased gemma-4 to repeat spawn_room. Warming the rooms up once up front and testing every case in a fresh turn (the live demo condition) gives the honest number — 76%, with spawn_room/add_agents/check_progress all solid.
kanban channel delete keeps the append-only log, so a room re-created with the same name replayed stale missions/joins from prior runs. Remove the .jsonl on reset for a true clean slate.
gemma-4-26b could not reliably route the five tools from descriptions or the
persona — it keys on salient words ('repo' -> spawn_room, 'room' -> add_agents)
and ignores the rest, and at its default warmth the same sentence routed
differently across runs. Two fixes:
- Session temperature 0, so a given utterance routes the same way every time.
- A deterministic intent safety net (tools/intent.ts): the room-operation and
show verbs ('tell/remind the room', 'N more / double the team', "how's it
going", 'show me / put on the screen') are distinctive enough to recognize
in code. The gateway runs it over the user's actual utterance and, when it
confidently reads a different tool than the model picked, overrides — else it
defers to the model (which nails spawn_room work and plain opens). The
utterance is captured from the turn transcript, the whisper shim, or a typed
turn, so it is set before the tool call lands.
Serial routing (one turn at a time, as in the live demo) is 15/15 on the
delegation set, up from 9/17. intent.unit.test.ts asserts the corrector on the
exact eval dataset — every steer/status/show utterance maps to the right tool,
and the six real-work requests are left untouched (returns null). CI runs it.
evaluation.run fires every case in parallel, which races the per-session utterance capture against the shared gateway and reported a concurrency artifact (53-71%). Gate the gateway hits through a one-at-a-time mutex (the live demo condition). With the intent corrector + temperature 0, serial = 17/17 = 100%.
The demo centerpiece is the swarm visibly self-organizing in the channel, so every agent MUST join it. Three things were losing agents: - A fixed 6s delay before sending the mission raced claude's cold boot; a slow agent got the keystrokes before its REPL was drawn and never ran 'kanban channel join'. Replaced with waitForClaudeReady (polls the pane). - Launching agents in parallel raced the kanban CLI and could collapse two agents onto one card. Split into two phases: create every session sequentially, then deliver missions as each claude becomes ready (the slow boots still overlap, so it stays fast). - Under load even a ready-looking REPL can swallow the first keystrokes, so after sending we confirm the agent's handle appears in the channel and re-send once if it didn't. Max also pins the mission before anyone joins, so it's in the history each agent reads on arrival. Engine unit tests unchanged and green.
focusChannel shelled 'kanban channel open' (a kanbancode:// deep link) to bring the board forward. When the URL scheme is registered to a different build than the one running (e.g. after a dev rebuild), the deep link relaunches that build and trips KanbanCode's quit-protection dialog mid-demo. Bring the already-running KanbanCode forward via System Events instead — it never launches anything, so no relaunch and no dialog. The top-left pin is unchanged. Channel auto-select rode on the deep link and is dropped here; a non-relaunching select belongs on the KanbanCode side.
Diagnosis: all agents boot and receive the mission (tens of k tokens of work),
but claude dives into the mission and doesn't reliably run 'kanban channel
join' itself — so only one agent ever appeared in the channel. The previous
confirm-and-resend made it worse, piling queued prompts onto busy agents.
Fix: right after 'kanban launch', the gateway force-joins the agent into the
channel by its returned card id ('kanban channel join --as <slug-handle>
--as-card-id <cardId>'). Membership is then guaranteed and independent of what
claude does; the card link means the agent still receives the channel's
broadcasts in its pane, and when it later posts from its tmux the membership
converges on the same handle (keyed by card id). The agent brief drops the
join step (it's already a member) and leads with read-history + claim-a-slice.
Verified: a 3-agent spawn now lands all three as tmx_aria/tmx_bolt/tmx_cleo
within ~8s.
Two notes from the live run: - No way to tear a room down by voice. Add a close_room tool (kill the dark mode room / shut down the room / stop the agents on X) that kills the agents' tmux sessions and deletes the channel, with a deterministic intent rule so gemma doesn't see 'room' and pick add_agents. - The KanbanCode board ended up UNDER the team's other Warp windows. The focus-last raise existed but was fire-and-forget and raced the equally fire-and-forget terminal pops. Now phase 1 collects the terminal-open promises, and the board raise is awaited AFTER they all settle, so it wins the z-order. Mission delivery moves to a background phase 2 (no window ops), so the board stays on top while claude boots.
Two rooms drew agent names from the same pool starting at aria/bolt/cleo, and the tmux slug seeds a deterministic sessionId, so a second room resumed the first room's claude under a new mission and its coordination was clobbered. Allocate names monotonically: take one only if it is neither already in use nor backed by a live tmux session (covers survivors of a gateway restart), and recycle the pool only once the whole fleet drains. close_room now really kills each session by slug (matching the reset script) and closes the leftover terminal window, so a freed name is safe to reuse and 'kill the room' actually clears the dead panes off screen.
open_url fired and Max said 'on the screen', but no Chrome appeared. The chromeless app instance quits when its last window closes, so its pid dies; openBrowser still held that pid and, on the same-url path, raised it to the front instead of respawning. Raising a dead pid is a silent no-op, so nothing showed. Reuse a tracked window only when its process is still alive (signal-0 check); otherwise drop the stale entry and spawn a fresh window.
Adds the close_room scenario to specs/room-orchestration.feature, a voice scenario test (kill the dark mode room -> close_room), and kill/close cases in the delegation eval. Adds the 2-minute meeting skit to DEMO.md with the exact spoken wording for the demo.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Hackathon build for the theme "what if compute and tokens were no longer a limitation".
A meeting room with an always-on microphone and a realtime voice agent (Max, on Inworld gemma-4). You talk; Max turns it into a swarm. A spoken mission becomes N Claude Code agents that all land in one shared Kanban Code channel and self-organize there: one claims the CSS foundations, another takes the pages, a third reviews. You assigned no roles. The channel is the agent loop, a natural ralph-loop where the agents hold each other to the mission while you keep talking.
It drives the real desktop, not a tab: real Warp terminals fly open with Claude Code working, the KanbanCode board comes forward on the room's channel so you watch them coordinate live, and
open_urlputs the running site or a GitHub issue on the screen. A floating HUD shows the room is listening, on top of everything.Why the swarm wins
Architecture
orchestrator/room.ts: spawn a swarm into a kanban channel, the coordinate-first agent brief, a deterministic project registry that resolves a spoken hint to a real checkout + URL with no LLM in the hot path), and the desktop control layer (platform-agnosticDesktopController; macOS adapter via osascript + Warp + chromeless Chrome; Windows stub).spawn_room(mission, topic, agents?, project?),message_room(steer a running room),add_agents(grow it),open_url(show a site / issue),check_progress(status from the channel).room-orchestration.featurecovers the swarm).The agents talk over Kanban Code channels: each agent has a tmux session kanban tracks, and
kanban channel sendbroadcasts into every other agent's pane. Max joins as@maxto drop the mission in and pass along voice corrections.Hard problems solved on the way
scripts/experiment-delegation.ts) showed gemma-4-26b mis-routing the room-op commands ("throw two more agents", "how's it going", "tell the room…") — it keys on salient words and ignores both the tool descriptions and the persona, and at its default warmth the same sentence routed differently across runs. A full description rewrite moved it 0%. The fix is two-part: sessiontemperature: 0(stable), plus a deterministic intent safety net (tools/intent.ts) that reads the user's actual utterance and overrides the model's pick when the verb is unambiguous, deferring to the model otherwise. Serial routing (one turn at a time, as in the live demo) went from 9/17 to 17/17;intent.unit.test.tsasserts the corrector on the exact eval dataset.kanban channel join. A fixed delay before sending the mission raced claude's cold boot (lost keystrokes → no join), and parallel launches raced the kanban CLI. Fixed by waiting for the REPL to be drawn, launching sessions sequentially then delivering missions as each boots, and confirming the handle appears in the channel (re-sending once if not).localhost:3000. The resolver trusts a real remote URL (a GitHub issue) but maps localhost guesses and bare phrases ("the website") to the project's real URL.config.ts, which requires live API keys at import; pure helpers (channel naming, the coordinate-first brief, the URL resolver, the intent corrector) moved into config-free modules so CI runs them with no.env.propertiesschemas (every tool keeps a required param), and doesn't transcribe committed audio underturn_detection: null(a gateway whisper STT shim injects the turn as text). Transient capacity errors retry then switch to a fallback model mid-session.Proof
scripts/experiment-delegation.ts) → LangWatch, scoring which tool each spoken request routes to.That unprompted teammate delegation, with no role assigned, is the whole thesis.
Screenshots
Captured during live QA runs (the desktop is the demo). Hero shots: the swarm of terminals fanning out with the KanbanCode board forward on the room channel; the channel showing agents claim slices and review each other; a browser window opened by
open_url.