Skip to content

docs(dev): verify tracing guide against code and remove migrated status (#206) #219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
antiv merged 3 commits into from
Oct 11, 2026
Merged

docs(dev): verify tracing guide against code and remove migrated status (#206) #219

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c82c93b
docs(dev): verify tracing guide against code and remove migrated stat…
Fire162 Oct 11, 2026
649d1a6
Address review: correct OTLP export nuances, tracer providers, and pr…
Fire162 Oct 11, 2026
4e91ee6
Merge branch 'main' into docs/tracing-clean
antiv Oct 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 16 additions & 11 deletions docs/dev/tracing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ title: "Tracing"
summary: Turn on OpenTelemetry tracing, view traces in the dashboard and export them.
audience: dev
order: 80
status: migrated
covers:
- shared/utils/tracing/**
- server/proxy_routes.py
---

# OpenTelemetry Distributed Tracing
Expand All @@ -21,24 +21,31 @@ MATE uses OpenTelemetry for distributed tracing of agent executions, LLM calls,
OTEL_TRACING_ENABLED=true

# Optional: OTLP endpoint for Jaeger, Grafana Tempo, Datadog, Honeycomb (OTLP over HTTP)
# When unset, only DB export runs (for dashboard)
# There is no default: when unset, spans only go to the database
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

# Service name for traces
# Service name for traces (only applies when MATE initializes its own TracerProvider)
OTEL_SERVICE_NAME=mate

# Store spans in database for dashboard viewer (default: true when tracing enabled)
OTEL_TRACES_DB_EXPORT=true
```

### Zero Performance When Disabled
### Performance When Disabled

When `OTEL_TRACING_ENABLED=false` (default), tracing is completely disabled. No overhead is incurred.
When `OTEL_TRACING_ENABLED=false` (default), tracing is disabled. Tool wrapping is skipped, but calls via `trace_context` still route through a no-op tracer.

## Architecture & Tracer Providers

- **TracerProvider integration**: When ADK has already initialized an SDK `TracerProvider`, MATE adds only its `BatchSpanProcessor(DatabaseSpanExporter())` to it and returns. In this case, MATE's OTLP exporter and `OTEL_SERVICE_NAME` configuration do not apply.
- **Standalone TracerProvider**: When no SDK provider is found, MATE initializes its own `TracerProvider` with `service.name` set to `OTEL_SERVICE_NAME`.
- **Database Span Exporter**: When `OTEL_TRACES_DB_EXPORT=true`, spans are batched (1-second flush delay) and persisted to the `trace_spans` table for visualization in the dashboard.
- **OTLP Exporter**: When `OTEL_EXPORTER_OTLP_ENDPOINT` is explicitly set, spans are exported using OTLP over HTTP (`opentelemetry.exporter.otlp.proto.http`). gRPC is not supported, and `OTEL_EXPORTER_OTLP_PROTOCOL` is not read.

## Instrumentation

| Instrumentation Point | Span Name | Attributes |
|----------------------|-----------|------------|
|---|---|---|
| LLM inference | `gen_ai.inference` | `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| RBAC check | `mate.rbac_check` | `mate.agent.name`, `mate.user.id`, `mate.rbac.allowed` |
| Tool invocation | `mate.tool` | `mate.tool.name`, `mate.agent.name` |
Expand All @@ -53,11 +60,9 @@ The `/dashboard/traces` page shows traces from the database when `OTEL_TRACES_DB

To export to Jaeger, Grafana Tempo, Datadog, or Honeycomb:

1. Run an OTLP-compatible collector (e.g. Jaeger all-in-one, Grafana OTLP collector)
2. Set `OTEL_EXPORTER_OTLP_ENDPOINT` to the collector's OTLP HTTP URL, for example
`http://localhost:4318`. There is no default: without it, spans only go to the
database. The exporter is OTLP over HTTP; gRPC is not supported.
1. Run an OTLP-compatible collector (e.g. Jaeger all-in-one, Grafana OTLP collector).
2. Set `OTEL_EXPORTER_OTLP_ENDPOINT` to the collector's OTLP HTTP URL (for example `http://localhost:4318`). There is no default: without this variable, spans only go to the database. Only OTLP over HTTP is supported (`opentelemetry.exporter.otlp.proto.http`). gRPC is not supported, and `OTEL_EXPORTER_OTLP_PROTOCOL` is ignored.

## W3C Trace Context

Trace context is propagated via `traceparent` and `tracestate` headers. The auth server proxy forwards requests to ADK with trace context injected when tracing is enabled.
Trace context is injected by the auth server proxy (`_inject_trace_headers` in `server/proxy_routes.py`) when forwarding requests to ADK if tracing is enabled. Nothing extracts trace context from inbound external requests.
Loading