Skip to content

feat: auto-reconnect backends instead of 404ing forever when down - #63

Open
FelixIsaac wants to merge 1 commit into
tbxark:masterfrom
FelixIsaac:felix/auto-reconnect
Open

feat: auto-reconnect backends instead of 404ing forever when down#63
FelixIsaac wants to merge 1 commit into
tbxark:masterfrom
FelixIsaac:felix/auto-reconnect

Conversation

@FelixIsaac

Copy link
Copy Markdown

Problem

If a backend is unreachable when the proxy starts (and panicIfInvalid is false, the default), its route is never registered and the endpoint returns 404 page not found forever — the only fix is restarting the whole proxy.

Root cause is in startHTTPServer (http.go): the route is registered inside the per-backend goroutine, only after addToMCPServer succeeds:

errorGroup.Go(func() error {
    addErr := mcpClient.addToMCPServer(ctx, info, server.mcpServer)
    if addErr != nil {
        if panicIfInvalid { return addErr }
        return nil            // <- gives up; httpMux.Handle never runs
    }
    ...
    httpMux.Handle(mcpRoute, ...) // <- only reached on success
})

Two real-world cases this hurts:

  1. Down at startup, comes up later — e.g. the proxy runs at login before an app that backs a streamable-http server is ready → route 404s permanently.
  2. Was up, then dropped — the app/subprocess restarts; the existing ping task (client.go) detects the failure but only logs it, never reconnects.

I'd been working around both with an external watchdog that restarts the whole proxy on a timer; this fixes it in-process instead.

Changes

  • Register the route up-front, before connecting. The handler resolves the live client lazily via a mutex-guarded getClient(), so the endpoint exists immediately and tools populate once the backend is reachable.
  • Retry the initial connect in the background (connectWithRetry) until it succeeds, instead of giving up after one attempt.
  • Reconnect on ping failure: rebuild the underlying client, re-initialize, and re-register tools/prompts/resources, swapping it in atomically. Transparent to in-flight/future requests since handlers go through getClient().
  • Enable ping for stdio backends too, so a crashed subprocess is detected and respawned (stdio previously had no liveness check).
  • Removed the now-unused errgroup usage.

New options (per-server, inheritable from mcpProxy.options)

Option Type Default Meaning
autoReconnect bool true Retry-at-startup + reconnect-on-drop. Set false to restore fail-once behaviour.
reconnectInterval duration (ns) 15s Gap between startup connection attempts.

panicIfInvalid still fails fast and takes precedence over autoReconnect.

Testing

Built and go vet clean. Verified end-to-end on Windows:

Down at startup (case 1): backend pointed at a closed port →

  • GET /deadbackend/mcp returns 200 (route registered), not 404
  • an unregistered route still 404s (control)
  • logs show Connecting (attempt N) retrying at the configured interval

Drop + recovery (case 2): real @modelcontextprotocol/server-everything over stdio →

  • connects, lists tools, serves traffic
  • killed the server subprocess → MCP Ping failed: transport closed (count=1) → re-listed tools → Reconnected after 1 ping failure(s) within one ping cycle
  • tools/call echo after reconnect returns Echo: alive-after-reconnect (new subprocess serving traffic)

Compatibility

Default behaviour change: backends now self-heal by default rather than failing once. Set autoReconnect: false to opt out. panicIfInvalid: true is unaffected.

Previously a backend that was unreachable at startup (with
panicIfInvalid:false) was silently skipped: its HTTP route was only
registered *after* a successful connect, so the endpoint 404'd forever
until the whole proxy was restarted. A backend that dropped later (e.g.
an app that backs a streamable-http server being restarted) was detected
by the ping task but never re-established.

This makes connections self-healing:

- Register each server's HTTP route up-front, before connecting. The
  handler resolves the live client lazily, so the endpoint exists
  immediately and tools appear once the backend is reachable.
- Retry the initial connect in the background until it succeeds
  (connectWithRetry), instead of giving up after one attempt.
- On ping failure, rebuild the underlying client, re-initialize and
  re-register tools/prompts/resources, swapping it in atomically.
- Enable the ping task for stdio backends too, so a crashed subprocess
  is detected and respawned (previously stdio had no liveness check).

New per-server options (inheritable from mcpProxy.options):
- autoReconnect (bool, default true) - set false to restore fail-once.
- reconnectInterval (duration, default 15s) - startup retry gap.

panicIfInvalid still fails fast as before. Tool handlers resolve the
client via a mutex-guarded getClient(), so reconnects are transparent to
in-flight and future requests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces an auto-reconnect mechanism for MCP clients, allowing the proxy to register HTTP routes up-front and lazily resolve backend connections. It adds configuration options for AutoReconnect and ReconnectInterval, background connection retries, and mutex-based synchronization to safely swap connections on failure. The review feedback highlights potential nil pointer panics in client.go if c.options or opts is nil, suggesting safe defaults instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread client.go
ticker := time.NewTicker(pingInterval)
defer ticker.Stop()

autoReconnect := c.options.AutoReconnect.OrElse(true)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If c.options is nil (which can happen if the client is constructed directly or in tests without going through the config loader), accessing c.options.AutoReconnect will cause a nil pointer panic. We should check if c.options is nil before accessing its fields.

	autoReconnect := true
	if c.options != nil {
		autoReconnect = c.options.AutoReconnect.OrElse(true)
	}

Comment thread client.go
Comment on lines +437 to +453
gap := opts.ReconnectInterval
if gap <= 0 {
gap = defaultReconnectGap
}
autoReconnect := opts.AutoReconnect.OrElse(true)

attempt := 0
for {
attempt++
log.Printf("<%s> Connecting (attempt %d)", c.name, attempt)
err := c.addToMCPServer(ctx, info, mcpServer)
if err == nil {
log.Printf("<%s> Connected", c.name)
return
}
log.Printf("<%s> Failed to connect: %v", c.name, err)
if opts.PanicIfInvalid.OrElse(false) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

If opts is nil, accessing opts.ReconnectInterval, opts.AutoReconnect, or opts.PanicIfInvalid will cause a nil pointer panic. We should check if opts is nil and use safe defaults.

	gap := defaultReconnectGap
	autoReconnect := true
	panicIfInvalid := false
	if opts != nil {
		if opts.ReconnectInterval > 0 {
			gap = opts.ReconnectInterval
		}
		autoReconnect = opts.AutoReconnect.OrElse(true)
		panicIfInvalid = opts.PanicIfInvalid.OrElse(false)
	}

	attempt := 0
	for {
		attempt++
		log.Printf("<%s> Connecting (attempt %d)", c.name, attempt)
		err := c.addToMCPServer(ctx, info, mcpServer)
		if err == nil {
			log.Printf("<%s> Connected", c.name)
			return
		}
		log.Printf("<%s> Failed to connect: %v", c.name, err)
		if panicIfInvalid {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant