Keeper background job has no metrics, alerting, or observability into failure trends

[abayomicornelius]

The keeper subsystem currently has two independent trigger paths that both call SubscriptionService::run_due_executions — the background loop in main.rs::run_keeper_loop (a tokio::time::interval on keeper_poll_interval_secs) and the manual POST /api/keeper/run-subscriptions endpoint (src/routes/keeper.rs, see also issue on its missing auth) — but neither path has any metrics, alerting, or structured observability beyond ad-hoc tracing::info!/tracing::warn! log lines:

Ok(summary) => {
    if summary.considered > 0 {
        tracing::info!(executed = summary.executed, failed = summary.failed, considered = summary.considered, "Keeper pass complete");
    }
}
Err(e) => { tracing::warn!(error = %e, "Keeper pass failed"); }

Why it matters — marked spike

This is an open-ended observability/reliability design question, not a single fixed patch, because it touches several interacting concerns that need a coherent design rather than independent point fixes:

• Silent degradation risk: if run_due_executions returns Err(AppError::KeeperUnavailable(...)) every single poll (e.g. because KEEPER_SECRET_KEY/SUBSCRIPTION_CONTRACT_ID were never configured, or the keeper account has run out of XLM to pay fees), the only signal is a repeating tracing::warn! log line — there's no counter, no alert threshold, and no way for an operator to notice "subscriptions have not executed in 6 hours" without actively tailing logs.
• No visibility into per-subscription failure trends: SubscriptionService::run_due_executions already tracks failure_count/last_error per subscription (marking it failed after MAX_CONSECUTIVE_FAILURES = 3) — but there's no aggregate view (e.g. "how many subscriptions are currently in a failing-but-not-yet-terminal state") exposed anywhere, meaning a systemic issue (e.g. Soroban RPC endpoint down) only becomes visible to an operator once individual subscriptions start hitting their 3-strike terminal failed state.
• Cross-references issue No graceful shutdown handling despite tokio-util 'full' dependency #14 (graceful shutdown spike) and the general lack of a /metrics endpoint anywhere in src/routes/mod.rs — there's no Prometheus-style counter/gauge infrastructure in this codebase at all today, so this issue is partly "should the project adopt a metrics crate" (a broader architectural decision) rather than "add three counters to keeper.rs."

Investigation questions

• Does this project want Prometheus-style /metrics (e.g. via the metrics/metrics-exporter-prometheus crates) as a general capability (benefiting HTTP request metrics too, not just the keeper), or is a lighter-weight approach (e.g. a periodic summary log plus a DB-queryable "keeper health" row) sufficient for current operational needs?
• Should keeper failures escalate beyond logs at some threshold (e.g. N consecutive failed passes, not just per-subscription failures) — and if so, via what channel (webhook, email, PagerDuty integration)? This is a product/ops decision, not purely technical.

Implementation sketch (once scope is settled)

• Minimal: track a last_successful_pass_at/consecutive_pass_failures in a small in-memory AppState field (or a dedicated DB row) updated by both trigger paths, and expose it via a GET /api/keeper/status admin-authenticated endpoint (reusing whatever admin-auth mechanism lands for the unauthenticated-keeper-endpoint issue).
• Fuller: adopt a metrics crate, emit keeper_pass_total{result="success|failure"}, keeper_subscriptions_executed_total, keeper_subscriptions_failed_total, and keeper_last_pass_timestamp gauges, scraped by whatever infra the deployment target already uses.

Testing strategy

• Once a status/metrics mechanism exists, test that N consecutive KeeperUnavailable errors correctly increment whatever "consecutive failure" counter is introduced, and that a subsequent success resets it — mirroring the existing failure_count reset-on-success logic already proven out in SubscriptionService::run_due_executions's per-subscription bookkeeping.