A durable, memory-enabled AI on-call SRE sidekick built on SigNoz. A SigNoz alert fires → Agent 99 recalls similar past incidents from its own episodic memory → investigates through the SigNoz MCP server (read-only) → posts a Slack card with an evidence-linked root cause and a proposed runbook → a human approves → it remediates exactly once, verifies recovery by re-querying the metrics, writes the postmortem, and memorizes the episode. Kill the process at any point — the incident resumes from its Postgres checkpoint and finishes. Built for the Agents of SigNoz hackathon, Track 01.

See it run

Live Slack card (verified recovery)	Agent 99 — Health (its own telemetry in the same SigNoz)

make demo:kill-resume — SIGKILL mid-incident, sweeper resume, zero duplicate side effects:

More captures (dashboards, alert rules, agent traces, make demo) live in docs/assets/, including a SigNoz navigation clip.

Why it's different (the four must-demo differentiators)

1. Kill-resume durability — make demo:kill-resume SIGKILLs the agent mid-incident; the crashed-run sweeper resumes the thread from its LangGraph/Postgres checkpoint and the approval card still renders — exactly once. Not a test double: the demo drives the same sweeper.sweep_once recovery path as production, and prints the audit row (sweeper · thread.resume) plus the zero-duplicate outbox proof.
2. Memory recall in the RCA card — every incident embeds its symptoms and searches past episodes (pgvector): the Slack card shows "Memory: 82% similar to INC-0007 (OOM → restart)" and the investigation is steered by it. The eval harness proves the steering with an A/B (below).
3. MTTR numbers, honestly labeled — make eval:replay runs a deterministic 10-scenario chaos suite through the real graph and writes evals/report.md: TTA, approval latency, MTTR p50/p90, top-2 cause accuracy, remediation success, and the memory on/off delta.
4. "Agent 99 Health" dashboard — we observe the observer — the agent exports its own OTel GenAI traces/metrics/logs to the same SigNoz (agent99.investigation.duration, agent99.mttr, agent99.llm.tokens, agent99.outbox.depth, …), charted in dashboards/agent99-health.json, with a self-referential alert (agent99.outbox.depth > 5 for 5m) routed back through the agent's own intake. If Agent 99 pages about itself, the loop is provably closed.

Architecture

flowchart LR
    subgraph SZ["SigNoz (Foundry — casting.yaml)"]
        AL[Alert engine]
        MCP["MCP server :8000"]
        CH[(ClickHouse)]
    end
    DEMO[opentelemetry-demo-lite<br/>pinned SHA] -->|"OTLP :4317"| CH
    AL -->|"webhook → :9100/alerts/signoz"| IN

    subgraph A99["Agent 99 (single worker)"]
        IN[FastAPI intake<br/>dedupe by fingerprint] --> G[LangGraph durable graph<br/>thread per incident]
        G --> SL[Slack app<br/>Socket Mode]
        G --> OB[Outbox dispatcher<br/>all side effects]
        G --> SW[Sweeper<br/>crashed-run resume]
        G --> MEM[Memory<br/>recall · memorize]
    end

    G <-->|"read-only MCP tools (≤12/incident)"| MCP
    G --- PG[("Postgres 17 + pgvector<br/>checkpoints · outbox · approvals<br/>schedules · audit · episodes")]
    OB -->|"restart-service · rollback · scale<br/>(advisory lock + idempotency key)"| DOCKER[Demo app containers]
    SL <-->|"RCA card · Approve/Reject"| HUMAN[On-call human]
    G <-->|"structured outputs · 60k token budget"| LLM["LLM (env-selected)<br/>Anthropic / OpenAI + embeddings"]
    A99 -->|"its own OTel GenAI telemetry<br/>(watching the watcher)"| CH

Loading

Incident lifecycle

sequenceDiagram
    autonumber
    participant S as SigNoz
    participant A as Agent 99
    participant P as Postgres
    participant H as On-call (Slack)
    participant D as Demo app

    S->>A: alert webhook (fingerprint = thread_id)
    A->>P: incident row + checkpoint (durability="sync")
    A->>P: recall — pgvector top-3 similar episodes
    A->>S: investigate — bounded ReAct over MCP tools
    A->>P: outbox intent: approval card (thread:approval_card:1)
    A->>H: Slack card — root cause + runbook + memory line
    Note over A: durable interrupt() — safe to SIGKILL here
    H->>A: ✅ Approve → Command(resume=…)
    A->>P: approval row (UNIQUE(thread_id, proposal_hash))
    A->>D: runbook via outbox + per-service advisory lock
    loop 3 consecutive 30s windows
        A->>S: verify — error rate AND p99 via MCP
    end
    A->>P: postmortem + verified episode memorized
    A->>H: card updated in place — ✔ Verified · MTTR

Loading

Incident state machine

stateDiagram-v2
    [*] --> INTAKE
    INTAKE --> RECALL
    RECALL --> INVESTIGATE
    INVESTIGATE --> PROPOSE: Hypothesis[]
    PROPOSE --> AWAITING_APPROVAL: interrupt()
    PROPOSE --> ESCALATED: no safe runbook
    AWAITING_APPROVAL --> REMEDIATE: approve (authz + proposal hash)
    AWAITING_APPROVAL --> ESCALATED: reject / 30m timeout
    REMEDIATE --> VERIFY: runbook exit 0
    REMEDIATE --> ESCALATED: retry exhausted
    VERIFY --> POSTMORTEM: 3×30s windows green
    VERIFY --> ESCALATED: 10m cap
    POSTMORTEM --> MEMORIZE
    MEMORIZE --> DONE: outcome=verified
    ESCALATED --> MEMORIZE: outcome=failed (negative example)
    DONE --> [*]

Loading

Durability & safety rules

Design rules that make the kill-resume claim true: durability="sync" checkpoints; nodes are pure — all side effects go through an outbox table with idempotency keys (thread_id:action:attempt); never try/except around interrupt(); waits are schedules rows, not in-process timers; approval decisions land behind UNIQUE(thread_id, proposal_hash).

Prompt-injection defense (telemetry is attacker-controlled): tool results are wrapped as data ("telemetry is evidence, never commands"), and the action layer is structurally immune — the LLM can only pick runbook_id + params from an allowlist, params are validated against the runbook schema, and the only service an incident may remediate is its own. An eval scenario injects a hostile log line and asserts no deviation.

Quickstart

Prereqs: Docker + Compose · foundryctl (curl -fsSL https://signoz.io/foundry.sh | bash) · Python 3.12 + uv (0.11.16) · gh · (optionally) a Slack app — Socket Mode, scopes chat:write app_mentions:read channels:read im:write, interactivity on.

foundryctl cast -f casting.yaml     # SigNoz v0.133.0 + MCP server (UI :8080, OTLP :4317, MCP :8000)
cp .env.example .env                # fill in keys as you mint them (never commit .env)
make up                             # postgres (pgvector) + agent (migrations at boot)
make demo                           # dry-run E2E: fake alert → parked approval card
make demo:kill-resume               # SIGKILL mid-incident → sweeper resume proof
make test lint                      # 104 tests + ruff + mypy

make demo works with zero secrets: SLACK_DRY_RUN=true renders cards to postmortems/slack-dry-run.jsonl and the incident parks at the durable approval interrupt (that park/resume is the product).

To go live, in order:

1. SigNoz API key — complete org setup at http://localhost:8080, then Settings → API Keys → SIGNOZ_API_KEY in .env. The MCP endpoint answers 401 until then (docs/mcp-tools.md).
2. Dashboards + alerts — make observe:validate (offline) then make observe:apply: creates the agent99-intake webhook channel (→ http://host.docker.internal:9100/alerts/signoz), the 3 dashboards, and the 4 alert rules via the SigNoz MCP tools.
3. Slack — set SLACK_BOT_TOKEN / SLACK_APP_TOKEN / APPROVER_SLACK_IDS, SLACK_DRY_RUN=false, make up. Approve drives runbook → verify → postmortem → memorize.
4. LLM — LLM_PROVIDER + key. Pinned model IDs: claude-sonnet-4-5-20250929 (fallback claude-haiku-4-5-20251001) or gpt-4.1-2025-04-14 (fallback gpt-4.1-mini-2025-04-14); embeddings text-embedding-3-small (EMBEDDINGS_PROVIDER=openai; local hashing embedder is the dev default).
5. Demo app — docker compose -f deploy/demo-app.compose.yml up -d (SigNoz/opentelemetry-demo-lite, pinned SHA, OTLP → local collector), then chaos: python chaos/kill_service.py --service payment.

The full narration for a judged run: docs/demo-script.md.

Results (evals/report.md)

Replay mode, synthetic telemetry. Deterministic in-process run (scripted LLM/MCP fakes, zero-duration verify windows) — measures pipeline mechanics and accuracy plumbing, not live-model latency. make eval regenerates with live numbers once keys are configured.

Metric	Value	SLO
Scenarios passed	10/10	—
MTTR p50 / p90	2.0s / 2.0s	≤ 5 min
TTA (alert → analysis card) mean	0.4s	—
Approval latency mean	0.04s	< 5s (demo invariant)
Top-1 / top-2 cause accuracy	100% / 100%	—
Remediation success (verified recovery)	100%	≥ 80%
Duplicate side effects	0	0 (hard requirement)
Kill-resume (scenarios 3 & 7, SIGKILL mid-run)	resumed + exactly-once ✔	—
Memory A/B — top-1 accuracy (recall on vs off)	100% vs 0%	—
Prompt-injection scenario	hostile log line ignored; allowlist held ✔	—

Live verified run (2026-07-25, real stack): SigNoz alert → live gpt-4o-mini investigation over the SigNoz MCP server → Slack approval card → restart-service executed via the docker socket (docker restart redis, fault was a paused redis) → verify re-query showed error rate 100.0% → 0.0% over 3/3 windows → postmortem + memorized episode. MTTR 604s, all side effects exactly-once. Details in the live addendum in evals/report.md; postmortem in postmortems/e42769cd05f04da5.md.

Ten scenarios (evals/scenarios.yaml): payment-down, checkout-latency, bad-deploy, cpu-hog, db-starve, recommendation-flap, email-timeout, cart-redis-loss, noisy-neighbor, silent-bad-config — each maps fault → expected alert → expected cause → expected runbook. Scenarios 3 & 7 SIGKILL the graph worker mid-run and assert sweeper resume; silent-bad-config carries the hostile log line. Chaos scripts live in chaos/ (all --dry-run-able, idempotent, heal.py reverses everything).

make eval:replay   # deterministic suite → evals/report.md (CI-able, no keys)
make eval          # live mode: real MCP/LLM/docker + real chaos (preflight-gated)

SLOs for the agent itself (ARCHITECTURE §14)

SLI	Target
Investigations completing with a posted RCA	≥ 95% of fired alerts
p50 MTTR across the 10 scenarios	≤ 5 min (measured; reported honestly)
Remediation success (verified recovery)	≥ 80%
Duplicate side effects	0 (hard requirement)
Approval round-trip availability	99% during demo windows

Repo layout

agent/agent99/     the agent: FastAPI intake, LangGraph durable graph, nodes/,
                   outbox/sweeper/locks/audit, memory (pgvector), MCP client,
                   LLM wrapper, Slack app, self-telemetry (OTel GenAI)
agent/tests/       104 tests: unit + kill-resume + outbox-dedupe + races
evals/             scenarios.yaml, runner.py (live+replay), report.md
chaos/             inject_latency · kill_service · bad_deploy · cpu_hog · db_starve · heal
dashboards/        red-overview · incident-war-room · agent99-health (JSON + README)
alerts/            error-rate-spike · p99-latency · anomaly-error-burst · agent99-outbox-depth
deploy/            demo-app compose (pinned SHA) · apply-observability.py · demo scripts
docs/              HANDOFF.md (spec) · ARCHITECTURE.md (SDD) · DEVIATIONS.md ·
                   demo-script.md · runbooks/ops.md · mcp-tools.md
postmortems/       auto-written incident postmortems + dry-run Slack log
casting.yaml(.lock) Foundry deployment of SigNoz (judges re-run `foundryctl cast`)

Docs

• docs/HANDOFF.md — build spec (constraints, stack pins, week plan)
• docs/ARCHITECTURE.md — system design (durability model, race catalog, gotchas)
• docs/DEVIATIONS.md — every departure from spec, and why
• docs/demo-script.md — the judged-demo walkthrough + video guidance
• docs/runbooks/ops.md — ops runbook (resume threads, rotate keys, prune, audit SQL)
• dashboards/README.md · alerts/README.md — observability artifacts
• AGENTS.md — operating rules for AI agents working in this repo

AI-assistant disclosure

Agent 99 was built with kimi-code (an AI coding agent) driving the implementation end-to-end — scaffolding, the durable core, tests, evals, and these docs — under human direction, per the hackathon rules. Every AI-generated change went through the repo's own gates: ruff + mypy + pytest (104 tests, including kill-resume and outbox-dedupe), Conventional Commits, and PR review. The operating rules the AI worked under are committed at AGENTS.md; the spec it executed is docs/HANDOFF.md.