feat(db): expose connection setup timeline

[ravarora2]

Why

Current pool metrics show checkout outcomes and pool state after the fact. They do not show when a checkout began or which writer-connection setup step failed. During startup and pool growth, operators need to tell pool saturation from a slow or unsafe connection setup.

What

• Count every instrumented database checkout when it starts, with a fixed operation label.
• Trace writer-pool creation and each new writer connection through connect, safety setup, isolation verification, and ready.
• Publish bounded metrics and exact lifecycle receipts without database URLs or raw errors.
• Document how to read these signals during a rollout or incident.

How

The existing typed checkout wrapper now records starts and current waiters. The huddle-history path uses that wrapper instead of a raw pool checkout.

The production SQLx after_connect hook records each writer connection with a process boot ID, connection ordinal, and ordered setup steps. Fixed enums keep metric cardinality bounded. A drop guard records cancellation once if setup exits before a terminal result.

Tests bind the production hook and cover initial minimum connections, later pool growth, drop-guard cancellation accounting, setup and isolation failures, Prometheus export, and relay boot output.

Risk

Medium. This changes the production writer-pool after_connect hook and adds one counter update to instrumented checkout paths. It does not change the database safety statements or their failure behavior. The main residual risk is telemetry overhead when SQLx creates a replacement connection; the records have a fixed schema and bounded labels.

Testing

Staging used pre-rebase head f58e9480a4f068db0c591f604fd6800fdd4bfc45. The deployed multi-architecture image came from GitHub Actions run 33780888255, manifest sha256:b9351fa644e08376cbe1999f9bee311d33d1799a68eadef7929c4f862a832fec.

The staging deployment brought both pods in ReplicaSet buzz-6c8758bd7d to Ready with zero restarts. Every observed initial writer connection reached db_ready, and the new metrics produced data in the rollout dashboard.

The rebase changed only the parent from 88687876f7808a2fd742b7eb2e4b9f87d999ad8d to current main; git range-diff shows the feature patch is unchanged.

Bigger picture

This is the database-timeline part of the startup and rollout observability work. The early-startup lifecycle foundation merged in #7258, so the rebase removed that duplicate commit from this PR.

Originating discussion: buzz://message?channel=6ac85131-70cd-4bda-a031-38d34114934e&id=fa2bed181c092697210a60bb6eedc55a665d5c1c6cabc1413a04686647011f71

Generated with Codex

[github-actions]

🔐 Codex Security Review

Status: review required for the current range.

The current range is f3408fc62045654183c704ceba2076b529a8140c...fb7ebf4bb621087e7192ded5819be95ecb83a2c1.
A new review must complete for this exact range. When manual authorization
is required, a Block organization member must comment exactly
@buzz-security-review fb7ebf4bb621087e7192ded5819be95ecb83a2c1 to authorize a new review.
Any previous review applies only to its recorded range.

[wesbillman]

Carl, an automated reviewer, commenting via Wes’s GitHub account.

No blocking findings at head 32d6e43176f7341104f39773e5e0e107200737f2 against base 44316ff72f5f7de014c66b01cbf534298a70c249. This is a review comment, not approval.

The composed production after_connect hook retains the safety SQL, bindings, and error propagation. Initial construction and later successful physical connections use the same instrumentation; setup failures stop before ready. Checkout starts preserve the existing terminal/waiter accounting. The exported ceilings reconcile: 198 operation-aware checkout series and 71 connection-step series, without sensitive or connection-ordinal labels.

Two non-blocking followups:

1. Clarify which pools the evidence covers. main.rs:38-47 constructs the audit pool without a lifecycle observer, while main.rs:259 supplies the observer for the main pool. Both emit the same writer metrics. Please clarify deploy/charts/buzz/README.md:217-221: with audit enabled, metric totals include main + audit pools, but lifecycle receipts describe only the main pool. They cannot be reconciled one-for-one.
2. Tighten production-path regression evidence and the testing claim. The new huddle acquire at crates/buzz-db/src/store/event.rs:240-244 is correctly wired, but its test at :2806-2842 asserts returned links, not telemetry; the operation-label matrix does not invoke this method. Add an isolated non-empty Db::huddle_started_links call asserting its own start/terminal deltas. Separately, connection cancellation is exercised by directly dropping a guard (connection_observability.rs:534-573), not by cancelling the actual connection-setup future. Narrow the PR’s cancellation-coverage claim or add that production-path case. Neither gap establishes a current runtime defect.

Validation: source/diff review of the production paths and SQLx 0.9 connection/retry behavior, integrated with three independent review lanes; clean worktree and git diff --check. Existing exact-head Rust, PostgreSQL, relay/backend integration, and Helm checks passed. No local suite or staging workflow was rerun. The reported staging deployment used pre-rebase head f58e9480a4f068db0c591f604fd6800fdd4bfc45, not this SHA.