Add retry/circuit-breaker resilience to the integration gateway route handlers

[Lakes41]

Difficulty: Advanced
Type: feature

Background
The "Local dashboard integration gateway" uses server-side Next.js route handlers to reach @guildpass/integration-client (an optional, private package) without exposing INTEGRATION_API_KEY to the browser, returning safe 503s if the package or key is missing, per the README.

Problem
Beyond the missing-package/key case, the gateway routes (/api/integration/membership, /api/integration/verify) have no described resilience against a slow or intermittently failing upstream integration client — a hanging or flaky call could tie up server resources or produce inconsistent user-facing errors.

Expected outcome
Both gateway routes apply a request timeout, a small number of retries with backoff for transient failures, and a simple circuit breaker that short-circuits to a fast 503 once repeated failures are detected, recovering automatically after a cooldown.

Suggested implementation

• Add a shared server-side utility (e.g., lib/integration/resilientCall.ts) implementing timeout + retry-with-backoff + a basic circuit breaker (open/half-open/closed states) usable by both route handlers.
• Wire both /api/integration/membership and /api/integration/verify through this utility.
• Ensure the existing "missing package/key → safe 503" behavior is preserved and distinguishable (e.g., via a response field or log message) from a circuit-breaker-triggered 503.
• Add tests simulating transient failures (retry succeeds), persistent failures (circuit opens), and recovery (circuit half-opens and closes again).

Acceptance criteria

• Transient upstream failures are retried transparently before surfacing an error
• Repeated failures trip a circuit breaker that fails fast instead of piling up slow requests
• Circuit recovers automatically after a cooldown period
• Existing missing-package/key 503 behavior is unaffected and distinguishable in logs

Likely affected files/directories

• app/api/integration/ (route handlers)
• lib/integration/ (new resilience utility)

[jchinedu]

Hi! I'd like to work on this issue.

I have experience building backend services with TypeScript and Node.js, including handling API integrations, error handling, and reusable service abstractions. This issue aligns well with my experience in improving the reliability of backend systems while keeping the implementation clean and maintainable.

My approach would be to introduce a shared resilience utility that encapsulates request timeouts, retry with exponential backoff, and a lightweight circuit breaker so both integration routes can share the same behavior. I'll make sure the existing missing package/API key handling remains unchanged and distinguishable from resilience-related failures, and I'll add tests covering transient failures, circuit opening, and automatic recovery after the cooldown period.

I'd be happy to contribute a well-tested implementation that follows the project's architecture and keeps the gateway resilient under intermittent upstream failures.

[grantfox-oss]

🦊 GrantFox — @jchinedu has been assigned to this issue as part of the Official Campaign | FWC26 campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #335)
2. Your PR will be reviewed by the Adamantine-guild maintainers

Good luck! Track your progress on GrantFox.