perf(channels): shard outbound dispatch per conversation#1464
Open
connermo wants to merge 1 commit into
Open
Conversation
A single dispatch goroutine delivered every reply in the process — every channel, every user — each Send blocking the next. One media upload of a few seconds froze all outbound traffic for that long, and sustained throughput was capped at one Send round-trip regardless of lane capacity. The ordering that loop actually protected is per-conversation: a run emits block replies, retry notices and a final answer to one ChatID, and those must arrive in that order. A global order was never required. Dispatch now shards on channel+ChatID. A conversation always maps to the same shard and is delivered serially there, so the ordering guarantee is unchanged, while unrelated conversations proceed in parallel. GOCLAW_OUTBOUND_SHARDS tunes the worker count (default 8). Temp media needed a real fix to go with it. Serial dispatch deduped it implicitly — the first delivery removed the file, and a later message carrying the same path found os.Stat failing and skipped it. Across concurrent shards that check alone is a TOCTOU race, so claims are now taken atomically and released after the send. Also makes the pool limits that bound this path configurable, defaults unchanged: GOCLAW_PG_MAX_OPEN_CONNS / GOCLAW_PG_MAX_IDLE_CONNS, and GOCLAW_HTTP_MAX_IDLE_CONNS / GOCLAW_HTTP_MAX_IDLE_CONNS_PER_HOST. The per-host limit binds when one provider takes nearly all traffic: every request past it pays a fresh TCP+TLS handshake, which a deployment raising LaneMain needs to be able to lift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Manager.dispatchOutboundran as a single goroutine callingchannel.Send()synchronously. Every reply in the process — every channel, every user — queued behind the slowest send.Two consequences:
LaneMaincapacity was configured.Why serialization was not required
The ordering that loop actually protected is per-conversation: a run emits block replies, a possible
Provider busy, retrying...notice, and the final answer to oneChatID, and those must arrive in that order. There are 41PublishOutboundcall sites, several of which target the same chat within one run.A global order was never needed. Ordering across different conversations carries no meaning.
Change
Dispatch now shards on
channel + ChatID:GOCLAW_OUTBOUND_SHARDStunes worker count (default 8).The channel name is part of the shard key because
ChatIDs are only unique within a channel.Temp media needed a real fix alongside it
Serial dispatch deduped temp media implicitly: the first delivery removed the file, and a later message carrying the same path found
os.Statfailing and skipped it. The existing comment ("already sent by another dispatch") states the assumption.Across concurrent shards that check alone is a TOCTOU race — two shards can both stat a live file and upload it twice. Claims are now taken atomically and released after the send. Non-temp (workspace) media passes through unclaimed, since it is never removed after delivery.
Also included
Pool limits that bound this same path, made configurable with defaults unchanged:
GOCLAW_PG_MAX_OPEN_CONNSGOCLAW_PG_MAX_IDLE_CONNSGOCLAW_HTTP_MAX_IDLE_CONNSGOCLAW_HTTP_MAX_IDLE_CONNS_PER_HOSTThe per-host limit binds when one provider takes nearly all traffic: every concurrent request past it pays a fresh TCP+TLS handshake instead of reusing a pooled connection. A deployment raising
LaneMainneeds to be able to lift it. Self-hosted / in-network providers have no reason to be throttled at 10.Tests
internal/channels/dispatch_shard_test.go:Sendis held hostage and the other must still be deliveredVerification
go build ./...,go build -tags sqliteonly ./...,go vet ./...— cleango test -racegreen forchannels,scheduler,store/pg,providersNote:
internal/channels/telegramTestDisplayWidthfails on this branch, but it fails identically on a pristineorigin/devworktree — pre-existing and unrelated.