feat(observability): thread the correlation id through every model-call, WebSocket and diagnostic site (Wave 3b of #3233) (#3237)

[oscharko]

Summary

Wave 3b of epic #3233 — the call-site half of Wave 3 (Wave 3a, #3241, shipped the gateway-layer contract). Every remaining production site in keiko-server that constructs a model request now passes logContext: { correlationId } from the request, run or job in scope, so a gateway retry, circuit-breaker transition or provider error line carries the same id as the BFF request that triggered it; WebSocket sessions resolve one id per connection; diagnostic sites that minted a fresh randomUUID() per failure thread the request/run/job id, or mint once per operation and set parentCorrelationId when the spawning request is known.

What ships

• 24 request-constructing model.call / callStream / gateway.chat sites in 18 files (chat send and regenerate, streaming, grounded QA folder/hybrid/multi-source, coding sidecar, QI runs, memory salience, …) pass logContext — pinned per site by tests that fail when the field is removed.
• WebSocket (voice-live-dictation, voice-realtime): one id resolved at handleUpgrade via resolveCorrelationId, used by every diagnostic on that connection.
• Background jobs: harness runs and workflow events spawned from a known request carry parentCorrelationId (shape-guarded; an out-of-shape value is dropped, never written under a marker).
• Diagnostic symmetry: the buffered chat path and grounded QA emit the same GatewayError diagnostic the streaming path emits (emitGatewayErrorDiagnostic); the remaining console.warn diagnostics in the chat-compaction path route through emitServerDiagnostic.
• Acceptance (correlation-threading.e2e.test.ts, against the real createUiServer handler): the same id on the http request line, gateway.chat.completed and the response header; a forced RateLimitError yields a diagnostic with that id and retryAfterMs; a scripted 429-then-200 fetch drives the real retry path under one id; two diagnostics on one WebSocket share the connection's id.

How this was built and checked

The Wave 3 implementation (nine work items, acceptance test, three adversarial reviewers, fixer, completion pass) was split for review load: 3a carried the contract and UI, this PR carries the 91 call-site files, re-applied onto dev after #3241 merged. Conflicts with 3a's review repairs (the diagnostic-writer sanitizer, the tool-bridge id mapping) were resolved with the tests as arbiter — no assertion weakened — and the op catalog regenerated.

Also in this PR: a correlation id that is honestly unknown is the shape-valid sentinel UNKNOWN_CORRELATION_ID ("unknown-correlation-id") at all eight fallback sites (gateway-error-diagnostic.ts, chat-handlers.ts, coding-sidecar-gateway.ts ×4, gateway-setup.ts, voice-handlers.ts) instead of the 7-character "unknown", which failed isValidCorrelationId and was rewritten by the diagnostic writer's sanitizer into its invalid-correlation-id marker — misreporting "no id available" as "malformed id". Regression test in coding-sidecar-gateway.test.ts fails against the bare literal.

Resolves #3237
Refs #3233

Scope

• In scope: the items above. Out of scope: persistence/security parity (feat(observability): persistence, indexing and security log-port parity; store fingerprints in the support bundle (#3239) #3244), HTTP/SSE detail and client diagnostics (Wave 5), analyzer/replay seed and docs (Wave 6).

Reuse And No-Duplication

• Existing Keiko functionality was inspected before implementation.
• This PR reuses, extends, generalizes, or consolidates existing functionality where practical (GatewayCallRequest.logContext, resolveCorrelationId, emitGatewayErrorDiagnostic, serverDiagnosticFromError).
• Any new implementation is limited to a documented capability gap in the linked issue.
• This PR does not introduce a parallel workspace, graph, relationship, policy, evidence, memory, connector, workflow, or UI subsystem where an existing subsystem can be extended.
• Refactoring or consolidation was considered when existing functionality was close but not shaped for this change.

Delivery Board

• Linked issue has a valid Parent Epic: #3233 and is attached as a GitHub sub-issue of it.
• Autonomous repair stayed on the PR branch; no direct dev push, force-push, finding dismissal, or required-gate bypass occurred.
• Native auto-merge armed only after the ADR-0135 direct required checks settled.

Product Impact

• Core generation engine (gateway observability; no behavioural change to calls or retries)
• Evidence, audit, or compliance artifact

Update Impact

• Release-impact metadata recorded here; catalog insertion is deferred to the release-cut PR per docs/release/release-impact-runbook.md.
• Release-note category: improvements.
• Priority: normal.
• User-visible change: every model call, retry and WebSocket diagnostic in the activity log carries the correlation id of the request, run or job that caused it; background jobs name their spawning request.
• Release-note bullet: Every model call and retry in Keiko's activity log now shares one correlation id with the request that caused it, and a background job names the request that spawned it.
• Supported-from versions: 0.2.0; first shipped in the next dev prerelease after 0.3.15.
• Affected state stores: <stateDir>/logs/server.log (additive fields).
• User action required and remediation: no-action-required.
• Release-owner review evidence: pending; bare issue reference Wave 3: Correlation threading end-to-end, gateway retry/usage/config observability, fetch-seam replay foundation #3237 until the release-cut PR.

Verification

Required:

• npm run gates:sonar was run on this diff and reported no finding.
• Required GitHub checks pass before merge.
• npm run check:ui-i18n — no UI production source changed.
• Local verification commands listed below.

Local verification:

npm run build:packages / typecheck / lint / format:check   -> PASS / PASS / PASS / PASS
npm run arch:check / arch:check:negative                  -> PASS / PASS
npm run check:error-observability                         -> PASS
npx vitest run packages/keiko-server                      -> 595 files passed, 10494 tests passed
npm test                                                  -> green except the coverage reality guard (closed by the regenerated baseline below)
npm run gates:sonar                                       -> PASS after one S7786 repair (TypeError after a type check in memory-audit-handler.ts)
test:coverage:packages / ui / scripts                     -> coverage collected; check:coverage:write-baseline PASS; every decreased floor restored (ADR-0158), no floor lowered
check:coverage:quality                                    -> release-target PASS; file-floors: 65 governed floor(s) checked; 0 violation(s)
npm run check:package-surface:assembled                   -> package-surface check passed
node scripts/generate-op-catalog.mjs                      -> regenerated (+ chat.turn.started); op-catalog-drift test green

• Core logic covered by unit, integration, property, or fixture tests (incl. the end-to-end correlation test).
• Security-sensitive change reviewed for trust boundaries (ids shape-guarded; no message text on any line).
• Supply-chain or package-surface change verified (check:package-surface:assembled).

Review And Closure

• The PR implements only the linked issue scope.
• Actionable review findings are fixed or explicitly dispositioned.
• Unresolved review threads are resolved before merge.

Risk Notes

• Pure threading of an existing optional field; every ModelPort/ChatModel implementation compiles unchanged.
• parentCorrelationId is additive; a top-level request never sets it.

[coderabbitai]

Warning

Your free Security trial is over. An organization admin can activate billing to continue.

[coderabbitai]

Warning

Review limit reached

@oscharko, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 14 minutes

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

Wait for the limit to reset, then comment @coderabbitai review or push new commits to the PR.

An organization admin can change what happens after included review limits in Billing.

How do review limits work?

CodeRabbit enforces per-developer PR review limits within each organization.

For paid Pro and Pro+ reviews, CodeRabbit uses a developer's included PR review attempts over the past 7 days to set the current hourly allowance. At typical activity levels, the full plan allowance applies. Higher sustained activity can lower the allowance until earlier attempts leave the 7-day window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5e96ce4e-b8d5-4346-ad5e-25a5eca6f1f9

📥 Commits

Reviewing files that changed from the base of the PR and between e55efd1 and 42acac5.

⛔ Files ignored due to path filters (1)

• docs/observability/op-catalog.generated.json is excluded by !**/*.generated.*

📒 Files selected for processing (86)

• docs/qa/package-coverage-baseline.json
• packages/keiko-server/src/atlassian/syncRoutes.test.ts
• packages/keiko-server/src/atlassian/syncRoutes.ts
• packages/keiko-server/src/atlassian/writeActionRoutes.test.ts
• packages/keiko-server/src/atlassian/writeActionRoutes.ts
• packages/keiko-server/src/browser-routes.test.ts
• packages/keiko-server/src/browser.ts
• packages/keiko-server/src/chat-compaction-evidence.test.ts
• packages/keiko-server/src/chat-compaction-evidence.ts
• packages/keiko-server/src/chat-compaction-model-summary.test.ts
• packages/keiko-server/src/chat-compaction-model-summary.ts
• packages/keiko-server/src/chat-handlers.test.ts
• packages/keiko-server/src/chat-handlers.ts
• packages/keiko-server/src/chat-stream-handlers.test.ts
• packages/keiko-server/src/chat-stream-handlers.ts
• packages/keiko-server/src/coding-context/codingContextRoutes.test.ts
• packages/keiko-server/src/coding-context/codingContextRoutes.ts
• packages/keiko-server/src/coding-sidecar-gateway.test.ts
• packages/keiko-server/src/coding-sidecar-gateway.ts
• packages/keiko-server/src/correlation-threading.e2e.test.ts
• packages/keiko-server/src/correlation.ts
• packages/keiko-server/src/desktop-chat-handlers.test.ts
• packages/keiko-server/src/editor/completionRoutes.ts
• packages/keiko-server/src/editor/inlineCompletionRoutes.ts
• packages/keiko-server/src/editor/localHistory/localHistoryCapture.test.ts
• packages/keiko-server/src/editor/localHistory/localHistoryCapture.ts
• packages/keiko-server/src/editor/patchApplyRoutes.test.ts
• packages/keiko-server/src/editor/patchApplyRoutes.ts
• packages/keiko-server/src/files.test.ts
• packages/keiko-server/src/files.ts
• packages/keiko-server/src/gateway-error-diagnostic.test.ts
• packages/keiko-server/src/gateway-error-diagnostic.ts
• packages/keiko-server/src/gateway-setup.test.ts
• packages/keiko-server/src/gateway-setup.ts
• packages/keiko-server/src/gitDelivery/mutationEvidenceLedger.test.ts
• packages/keiko-server/src/gitDelivery/mutationEvidenceLedger.ts
• packages/keiko-server/src/gitDelivery/syncEvidence.test.ts
• packages/keiko-server/src/gitDelivery/syncEvidence.ts
• packages/keiko-server/src/grounded-entailment-judge.test.ts
• packages/keiko-server/src/grounded-entailment-judge.ts
• packages/keiko-server/src/grounded-entailment-stage.ts
• packages/keiko-server/src/grounded-qa-hybrid.test.ts
• packages/keiko-server/src/grounded-qa-hybrid.ts
• packages/keiko-server/src/grounded-qa-multi-source.test.ts
• packages/keiko-server/src/grounded-qa-multi-source.ts
• packages/keiko-server/src/grounded-qa.test.ts
• packages/keiko-server/src/grounded-qa.ts
• packages/keiko-server/src/local-knowledge-grounded-qa.ts
• packages/keiko-server/src/memory-audit-handler.test.ts
• packages/keiko-server/src/memory-audit-handler.ts
• packages/keiko-server/src/memory-conflict-advisory.test.ts
• packages/keiko-server/src/memory-conflict-advisory.ts
• packages/keiko-server/src/memory-maintenance-handlers.test.ts
• packages/keiko-server/src/memory-maintenance-handlers.ts
• packages/keiko-server/src/memory-salience.test.ts
• packages/keiko-server/src/memory-salience.ts
• packages/keiko-server/src/qualityIntelligence/__tests__/figmaSnapshotAdapter.test.ts
• packages/keiko-server/src/qualityIntelligence/__tests__/generationPort.test.ts
• packages/keiko-server/src/qualityIntelligence/__tests__/judgePort.test.ts
• packages/keiko-server/src/qualityIntelligence/figmaSnapshotAdapter.ts
• packages/keiko-server/src/qualityIntelligence/generationPort.ts
• packages/keiko-server/src/qualityIntelligence/handoffRoutes.ts
• packages/keiko-server/src/qualityIntelligence/judgePort.ts
• packages/keiko-server/src/qualityIntelligence/modelPolicyRoutes.ts
• packages/keiko-server/src/qualityIntelligence/reCheckRoutes.ts
• packages/keiko-server/src/qualityIntelligence/runExecution.ts
• packages/keiko-server/src/qualityIntelligence/runRoutes.ts
• packages/keiko-server/src/run-engine.test.ts
• packages/keiko-server/src/run-engine.ts
• packages/keiko-server/src/run-handlers.test.ts
• packages/keiko-server/src/run-handlers.ts
• packages/keiko-server/src/sse-write.test.ts
• packages/keiko-server/src/sse-write.ts
• packages/keiko-server/src/terminal-routes.test.ts
• packages/keiko-server/src/terminal-routes.ts
• packages/keiko-server/src/update-remediation-routes.test.ts
• packages/keiko-server/src/update-remediation-routes.ts
• packages/keiko-server/src/update-remediation.test.ts
• packages/keiko-server/src/update-remediation.ts
• packages/keiko-server/src/voice-control-ws.test.ts
• packages/keiko-server/src/voice-handlers.ts
• packages/keiko-server/src/voice-live-dictation.ts
• packages/keiko-server/src/voice-realtime.test.ts
• packages/keiko-server/src/voice-realtime.ts
• packages/keiko-server/src/workspace-index-provider.test.ts
• packages/keiko-server/src/workspace-index-provider.ts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.8% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud

[oscharko]

Superseded by #3247, which now integrates this branch together with the other open logging-epic branches (Wave 3b, Wave 4a incl. all 16 review repairs, Wave 5, reviewer-audit repairs) into one pull request at the repository owner's request — each separate merge restarted the full gate matrix on the others. This branch's commits are merged into #3247 unchanged (42acac5b); review threads already answered here remain answered there.