feat: auto-reconnect backends instead of 404ing forever when down - #63
feat: auto-reconnect backends instead of 404ing forever when down#63FelixIsaac wants to merge 1 commit into
Conversation
Previously a backend that was unreachable at startup (with panicIfInvalid:false) was silently skipped: its HTTP route was only registered *after* a successful connect, so the endpoint 404'd forever until the whole proxy was restarted. A backend that dropped later (e.g. an app that backs a streamable-http server being restarted) was detected by the ping task but never re-established. This makes connections self-healing: - Register each server's HTTP route up-front, before connecting. The handler resolves the live client lazily, so the endpoint exists immediately and tools appear once the backend is reachable. - Retry the initial connect in the background until it succeeds (connectWithRetry), instead of giving up after one attempt. - On ping failure, rebuild the underlying client, re-initialize and re-register tools/prompts/resources, swapping it in atomically. - Enable the ping task for stdio backends too, so a crashed subprocess is detected and respawned (previously stdio had no liveness check). New per-server options (inheritable from mcpProxy.options): - autoReconnect (bool, default true) - set false to restore fail-once. - reconnectInterval (duration, default 15s) - startup retry gap. panicIfInvalid still fails fast as before. Tool handlers resolve the client via a mutex-guarded getClient(), so reconnects are transparent to in-flight and future requests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces an auto-reconnect mechanism for MCP clients, allowing the proxy to register HTTP routes up-front and lazily resolve backend connections. It adds configuration options for AutoReconnect and ReconnectInterval, background connection retries, and mutex-based synchronization to safely swap connections on failure. The review feedback highlights potential nil pointer panics in client.go if c.options or opts is nil, suggesting safe defaults instead.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| ticker := time.NewTicker(pingInterval) | ||
| defer ticker.Stop() | ||
|
|
||
| autoReconnect := c.options.AutoReconnect.OrElse(true) |
There was a problem hiding this comment.
If c.options is nil (which can happen if the client is constructed directly or in tests without going through the config loader), accessing c.options.AutoReconnect will cause a nil pointer panic. We should check if c.options is nil before accessing its fields.
autoReconnect := true
if c.options != nil {
autoReconnect = c.options.AutoReconnect.OrElse(true)
}| gap := opts.ReconnectInterval | ||
| if gap <= 0 { | ||
| gap = defaultReconnectGap | ||
| } | ||
| autoReconnect := opts.AutoReconnect.OrElse(true) | ||
|
|
||
| attempt := 0 | ||
| for { | ||
| attempt++ | ||
| log.Printf("<%s> Connecting (attempt %d)", c.name, attempt) | ||
| err := c.addToMCPServer(ctx, info, mcpServer) | ||
| if err == nil { | ||
| log.Printf("<%s> Connected", c.name) | ||
| return | ||
| } | ||
| log.Printf("<%s> Failed to connect: %v", c.name, err) | ||
| if opts.PanicIfInvalid.OrElse(false) { |
There was a problem hiding this comment.
If opts is nil, accessing opts.ReconnectInterval, opts.AutoReconnect, or opts.PanicIfInvalid will cause a nil pointer panic. We should check if opts is nil and use safe defaults.
gap := defaultReconnectGap
autoReconnect := true
panicIfInvalid := false
if opts != nil {
if opts.ReconnectInterval > 0 {
gap = opts.ReconnectInterval
}
autoReconnect = opts.AutoReconnect.OrElse(true)
panicIfInvalid = opts.PanicIfInvalid.OrElse(false)
}
attempt := 0
for {
attempt++
log.Printf("<%s> Connecting (attempt %d)", c.name, attempt)
err := c.addToMCPServer(ctx, info, mcpServer)
if err == nil {
log.Printf("<%s> Connected", c.name)
return
}
log.Printf("<%s> Failed to connect: %v", c.name, err)
if panicIfInvalid {
Problem
If a backend is unreachable when the proxy starts (and
panicIfInvalidisfalse, the default), its route is never registered and the endpoint returns404 page not foundforever — the only fix is restarting the whole proxy.Root cause is in
startHTTPServer(http.go): the route is registered inside the per-backend goroutine, only afteraddToMCPServersucceeds:Two real-world cases this hurts:
streamable-httpserver is ready → route 404s permanently.client.go) detects the failure but only logs it, never reconnects.I'd been working around both with an external watchdog that restarts the whole proxy on a timer; this fixes it in-process instead.
Changes
getClient(), so the endpoint exists immediately and tools populate once the backend is reachable.connectWithRetry) until it succeeds, instead of giving up after one attempt.getClient().errgroupusage.New options (per-server, inheritable from
mcpProxy.options)autoReconnecttruefalseto restore fail-once behaviour.reconnectInterval15spanicIfInvalidstill fails fast and takes precedence overautoReconnect.Testing
Built and
go vetclean. Verified end-to-end on Windows:Down at startup (case 1): backend pointed at a closed port →
GET /deadbackend/mcpreturns 200 (route registered), not 404Connecting (attempt N)retrying at the configured intervalDrop + recovery (case 2): real
@modelcontextprotocol/server-everythingover stdio →MCP Ping failed: transport closed (count=1)→ re-listed tools →Reconnected after 1 ping failure(s)within one ping cycletools/call echoafter reconnect returnsEcho: alive-after-reconnect(new subprocess serving traffic)Compatibility
Default behaviour change: backends now self-heal by default rather than failing once. Set
autoReconnect: falseto opt out.panicIfInvalid: trueis unaffected.