feat(testing): add observability smoke checks for metrics, logs, traces, and dashboards

[AdityaShome]

Summary

This PR strengthens the observability validation path by fixing metrics exporter startup in the monitoring dashboard example and hardening smoke checks for clearer diagnostics and deterministic Jaeger trace validation.

It keeps the current demo dashboard behavior, but ensures /metrics is reliably populated and makes the stack checks easier to debug when something is not ready.

Execution Flow

flowchart TD
    A["Start monitoring app"]
    B["Start observability stack"]
    C["Run smoke_check.sh"]

    A --> A1["Prime and start exporter"]
    A1 --> A2["metrics endpoint ready"]

    B --> B1["Prometheus healthy"]
    B --> B2["Loki ready"]
    B --> B3["Jaeger ready"]

    C --> C1["Check metrics families"]
    C --> C2["Check Prometheus queries"]
    C --> C3["Check Loki logs"]
    C --> C4["Inject and verify Jaeger trace"]

    A2 --> C1
    B1 --> C2
    B2 --> C3
    B3 --> C4

    C1 --> D["Smoke check passed"]
    C2 --> D
    C3 --> D
    C4 --> D

Loading

What Changed

1) Prime/start Prometheus exporter in dashboard example

• File: examples/monitoring_dashboard/src/main.rs
• After build_router(), the example now:
  • calls refresh_once().await to prime the cached metrics payload
  • starts the exporter refresh worker
• This prevents empty /metrics output during startup and improves scrape reliability.

2) Enable Zipkin collector explicitly in Jaeger service

• File: ops/observability/docker-compose.yml
• Added:
  • COLLECTOR_ZIPKIN_HOST_PORT=:9411
• This ensures the smoke script’s trace injection path (/api/v2/spans) is available consistently.

3) Improve smoke check observability and failure messaging

• File: ops/observability/smoke_check.sh
• Added clearer step-level output:
  • per-service readiness labels
  • elapsed wait-time progress
  • explicit “ok” confirmations for query checks
• Improved Zipkin injection error handling:
  • clearer actionable message when Jaeger Zipkin ingest is unavailable.

Validation Performed

• cargo check --manifest-path examples/monitoring_dashboard/Cargo.toml passed.
• bash -n ops/observability/smoke_check.sh passed.
• End-to-end run succeeded:
  • service health checks
  • /metrics family checks
  • Prometheus queries
  • Loki ingestion query
  • Jaeger service + trace visibility after injected smoke trace

Notes

• This PR does not change telemetry semantics or metric schema.
• Dashboard tested data source in examples/monitoring_dashboard/src/main.rs.
• Stack/services/log/trace plumbing is real and validated end-to-end.