Alerting stack: Grafana alerts, synthetic-check script, ONCALL.md runbook

[truthixify]

Tier: M (2-4 days) | Type: ops

Context. stellar/scripts/metrics-indexer/ ships a Prometheus scraper and a pre-built grafana-dashboard.json, and stellar/scripts/keeper/ runs the TTL keeper, but there is no alert layer on top of them. stellar/MAINNET_READINESS.md has Ops → Incident Response Runbook and Ops → On-call Rotation both unchecked. stellar/DEPLOYMENT.md contains zero references to monitor, on-call, alert, runbook, or incident. Wave 8 turns the metrics into a real alerting stack before mainnet.

Scope.

• Add Grafana alert rules alongside the dashboard: rate spike on send_count, drop on resolve_hit_count, missed keeper heartbeat, name-registration failure spike.
• Add stellar/scripts/synthetic-check.sh: end-to-end announcement / resolution / name-lookup ping against a target network, with pass / fail exit codes suitable for a cron.
• Publish stellar/ONCALL.md: escalation ladder, paging targets, incident-response checklist (declare, contain, communicate, resolve, postmortem), links to the pause and multisig runbooks.
• Update stellar/DEPLOYMENT.md with a new "Post-deploy monitoring" section pointing at all three.

Acceptance.

• Grafana alert rules JSON committed under stellar/scripts/metrics-indexer/ and loaded by the docker-compose in that dir
• synthetic-check.sh returns nonzero on injected failure and zero on happy path (verified against futurenet in CI)
• stellar/ONCALL.md reviewed and linked from stellar/README.md operations section
• stellar/MAINNET_READINESS.md Incident Response Runbook row checked with a link to ONCALL.md

Files. stellar/scripts/metrics-indexer/alerts.json (new), stellar/scripts/metrics-indexer/docker-compose.yml (or existing config file), stellar/scripts/synthetic-check.sh (new), stellar/ONCALL.md (new), stellar/DEPLOYMENT.md, stellar/MAINNET_READINESS.md, stellar/README.md.

[sudo-robi]

@sudo-robi has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi maintainer I'm a full stack developer and smart contract developer and auditor let me help you resolve this issue and I assure you it will be thoroughly done neatly as I have seen my way to solve the issue.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @sudo-robi to this issue.

[zinodict121]

@zinodict121 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi maintainer, I would like to work on this issue

ℹ️ Repo Maintainers: To accept this application, review their application or assign @zinodict121 to this issue.

[moon-code12]

@moon-code12 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi maintainer i would love to assign to this issue as a back end developer and I would be glad if it is assigned to me

ℹ️ Repo Maintainers: To accept this application, review their application or assign @moon-code12 to this issue.

[oluwadindodosimilenuotunfatasi]

@oluwadindodosimilenuotunfatasi has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hello! I've reviewed this issue in detail and I'm confident I can deliver a solution that meets the acceptance criteria. I have relevant experience with similar tasks and would welcome the opportunity to work on it.

ℹ️ Repo Maintainers: To accept this application, review their application or assign @oluwadindodosimilenuotunfatasi to this issue.

[ryzen-xp]

@ryzen-xp has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

Hi maintainers 👋
I’m a blockchain developer with strong expertise in Stellar, Rust,TS and Stellar SDK integrations. I’ve worked on multiple Stellar-based smart contract and backend systems.

I understand the scope of this issue and can deliver a reliable, production-quality solution with proper tests and documentation.
Plz assign me !!

ETA: 24 hours

ℹ️ Repo Maintainers: To accept this application, review their application or assign @ryzen-xp to this issue.

[Chidan09]

@Chidan09 has applied to work on this issue as part of the Stellar Wave Program's 8th wave.

​Hey Maintainer! Read through the issue details and I'm ready to dive in. Feel free to assign this to me.
​Checked out the requirements and acceptance criteria—I can have a fix ready that matches the design specs.
​Hi there, I took a look at the requirements and can pull together a clean fix. Would love to have this assigned to me.
​Hi Maintainer, interested in taking this on! I've done similar work before and can get started right away.
​Hello! I'd love the chance to tackle this issue and contribute to the project. Thanks!
​Hi, read through the ticket and I'm confident I can ship a solid, efficient solution for this.
​Hey Maintainer, happy to take ownership of this task and submit a solid PR once it's wrapped up.
​Reviewed the scope thoroughly—I can handle the fix while making sure all acceptance criteria are fully met.
​Hello, this aligns really well with my skillset. I'm ready to resolve it whenever you're open to assigning it my way.
​Hi Maintainer! Would love to contribute here. I can get a clean solution put together in short order.
​Hello, I reviewed the issue and clear on the target outcome. Happy to jump in and get to work on it!

ℹ️ Repo Maintainers: To accept this application, review their application or assign @Chidan09 to this issue.