feat: Add PostgreSQL metrics instrumentation via OpenTelemetry #127
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add comprehensive PostgreSQL connection pool and query metrics using OpenTelemetry SDK with OTLP export. Metrics are only collected when OTEL_EXPORTER_OTLP_ENDPOINT is configured. Metrics added: - db.client.connection.count (by state: idle/used) - db.client.connection.max, overflow.current, idle.max - db.client.connection.created_total, invalidated_total - db.client.connection.use_time (histogram) - db.client.operation.duration (histogram) - db.client.operation.slow_total (>500ms threshold) - db.client.operation.errors_total - db.client.response.returned_rows (histogram) - db.client.connection.health (1=healthy, 0=unhealthy) - db.client.replica.lag (for read replicas) Configuration: - OTEL_EXPORTER_OTLP_ENDPOINT: OTLP endpoint (required to enable) - POSTGRES_SLOW_QUERY_THRESHOLD: Slow query threshold in seconds (default: 0.5)
- Add OTel Collector service to docker-compose.yml with Prometheus exporter - Create otel/otel-collector-config.yaml with OTLP receiver and debug/prometheus exporters - Configure health_check extension for collector health monitoring - Fix pool.overflow() handling in db_metrics.py - use max(0, overflow) since SQLAlchemy returns negative values relative to max_overflow - Remove overflow from used count calculation to avoid double-counting
- Add warning-level logging to database error handler to capture actual exception messages, types, and truncated SQL statements - Add service.name attribute to all PostgreSQL metrics for better filtering in Grafana dashboards - Pass OTEL_SERVICE_NAME environment variable to metrics collector This change helps diagnose database errors that were previously only captured as metrics without detailed logging.
Aurora PostgreSQL doesn't support pg_last_xact_replay_timestamp() as it uses a different replication mechanism. This was causing continuous errors on the readonly pool. Removed: - db.client.replica.lag metric and histogram - check_replica_lag() method - is_replica parameter from metrics registration
Add dual export of PostgreSQL metrics to both OTel (Grafana/Mimir) and Datadog (StatsD). This provides visibility in both observability platforms. New StatsD metrics exported: - db.client.connection.count (gauge, by state) - db.client.connection.max (gauge) - db.client.connection.overflow (gauge) - db.client.connection.created (counter) - db.client.connection.use_time (histogram, ms) - db.client.connection.invalidated (counter) - db.client.operation.duration (histogram, ms) - db.client.operation.slow (counter) - db.client.operation.errors (counter) - db.client.response.returned_rows (histogram) - db.client.connection.health (gauge) - db.client.connection.health_check_failures (counter) Tags: service, db_system, pool, server, db_name, env, operation, table
RoxyFarhad
reviewed
Jan 15, 2026
| environment = self.environment_variables.ENVIRONMENT | ||
| service_name = os.environ.get("OTEL_SERVICE_NAME", "agentex") | ||
|
|
||
| if self.database_async_read_write_engine: |
Collaborator
There was a problem hiding this comment.
why do we need each of these engines?
RoxyFarhad
reviewed
Jan 15, 2026
| logger = make_logger(__name__) | ||
|
|
||
|
|
||
| def configure_statsd(): |
Collaborator
There was a problem hiding this comment.
wait sorry, why are we removing datadog here? don't we want the option to have it?
Collaborator
Author
There was a problem hiding this comment.
yes, whoops. this was not supposed to be getting removed
RoxyFarhad
approved these changes
Jan 15, 2026
Collaborator
RoxyFarhad
left a comment
RoxyFarhad
left a comment
There was a problem hiding this comment.
approving if you just add the DD metrics
RoxyFarhad
approved these changes
Jan 15, 2026
Fixes AttributeError when OTel is disabled but StatsD metrics are enabled. The base_attributes dict is now set before the early return, matching the pattern used by PostgresQueryMetrics and PostgresHealthMetrics.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
OTEL_EXPORTER_OTLP_ENDPOINTis configuredDD_AGENT_HOSTis availableMetrics Exported
Connection Pool (periodic, every 30s)
db.client.connection.countdb.client.connection.maxdb.client.connection.overflowConnection Events (on event)
db.client.connection.createddb.client.connection.use_timedb.client.connection.invalidatedQuery Performance (on every query)
db.client.operation.durationdb.client.operation.slowdb.client.operation.errorsdb.client.response.returned_rowsHealth (periodic, every 30s)
db.client.connection.healthdb.client.connection.health_check_failuresTags/Attributes
service/service.name: agentexdb_system/db.system.name: postgresqlpool/db.client.connection.pool.name: main, middleware, readonlyserver/server.address: database hostnamedb_name/db.namespace: database nameenv/deployment.environment: staging, productionoperation/db.operation.name: SELECT, INSERT, UPDATE, DELETEtable/db.collection.name: table nameConfiguration
OTEL_EXPORTER_OTLP_ENDPOINTDD_AGENT_HOSTPOSTGRES_SLOW_QUERY_THRESHOLDTest plan