Cumulative engine metrics are replayed and gauges duplicated across config-version series

[mwisner]

Component(s)

router

Component version

Router v0.336.0 (b9dd6b20d1d348188b18f53a6a41ba4b2a6c0bc7)

wgc version

N/A

controlplane version

Cosmo Cloud

router version

0.336.0

What happened?

Description

When OTLP subscription engine statistics are enabled, router.engine.messages.sent can replay its process-lifetime cumulative value as the first value of newly appearing wg.router.config.version metric series.

This creates a very large synthetic count spike even though incoming stream traffic and the actual number of active subscriptions remain stable. Active engine gauges such as router.engine.subscriptions and router.engine.triggers can also be duplicated across concurrent config-version series.

The exact lifecycle trigger has not yet been isolated. The observed series boundary coincided with two control-plane execution-config changes, but local router-config file watching/hot reloading was not enabled. The config-version transition and graph-server observer lifecycle are therefore evidence and a working hypothesis, not a confirmed prerequisite or root cause.

Observed evidence from one 30-second interval:

Series	router.engine.messages.sent
Existing config-version series	4,534
First new config-version series	2,161,482
Second new config-version series	2,161,482
Aggregated total	4,327,498

The two new series each began with the same accumulated process-lifetime counter value. In that interval, incoming stream messages peaked below 100 and the actual subscription count per config-version series remained approximately 203–206, so the 4.3M value was not real delivery volume.

The subscription gauge showed the same multi-series behavior: approximately 204 active subscriptions were reported under three config-version series, producing an aggregated value of approximately 613 even though there were still only approximately 204 active subscriptions.

At the metric-series transition, logs contained these messages 11 seconds apart, without a process restart:

17:13:07 Fetched router config
17:13:07 Router execution config has changed, hot reloading server
17:13:18 Fetched router config
17:13:18 Router execution config has changed, hot reloading server

These are Cosmo's execution-config lifecycle messages. They do not indicate that local router configuration file watching was enabled.

Relevant code

router.engine.messages.sent is defined as an Int64ObservableCounter:

cosmo/router/pkg/metric/engine_metrics.go

Lines 142 to 143 in b9dd6b2

	messagesSent, err = m.Int64ObservableCounter(engineMessagesSentKey,
	otelmetric.WithDescription("Number of subscription updates in the engine."))

The callback reads the shared engine report and observes the lifetime report.MessagesSent value with baseAttributes:

cosmo/router/pkg/metric/engine_metrics.go

Lines 173 to 202 in b9dd6b2

	func (e *EngineMetrics) registerObservers(stats statistics.EngineStatistics) error {
	instrumentList := e.instruments.toList()
	// Nothing to register
	if len(instrumentList) == 0 {
	return nil
	}
	rc, err := e.meter.RegisterCallback(func(_ context.Context, o otelmetric.Observer) error {
	e.observeInstruments(o, stats)
	return nil
	}, instrumentList...)
	if err != nil {
	return err
	}
	e.instrumentRegistrations = append(e.instrumentRegistrations, rc)
	return nil
	}
	func (e *EngineMetrics) observeInstruments(o otelmetric.Observer, stats statistics.EngineStatistics) {
	report := stats.GetReport()
	if e.instruments.connectionCount != nil {
	o.ObserveInt64(e.instruments.connectionCount, int64(report.Connections), otelmetric.WithAttributes(e.baseAttributes...))
	o.ObserveInt64(e.instruments.subscriptionCount, int64(report.Subscriptions), otelmetric.WithAttributes(e.baseAttributes...))
	o.ObserveInt64(e.instruments.triggerCount, int64(report.Triggers), otelmetric.WithAttributes(e.baseAttributes...))
	o.ObserveInt64(e.instruments.messagesSent, int64(report.MessagesSent), otelmetric.WithAttributes(e.baseAttributes...))

Those base metric attributes include wg.router.config.version:

cosmo/router/core/graph_server.go

Lines 257 to 269 in b9dd6b2

	s.baseOtelAttributes = baseOtelAttributes
	baseDefaultMuxAttributes := append([]attribute.KeyValue{otel.WgRouterConfigVersion.String(s.baseRouterConfigVersion)}, baseOtelAttributes...)
	mapper := newAttributeMapper(!rmetric.IsUsingDefaultCloudExporter(s.metricConfig), s.metricConfig.Attributes)
	mappedMetricAttributes := mapper.mapAttributes(baseDefaultMuxAttributes)
	if s.metricConfig.OpenTelemetry.RouterRuntime {
	// We track runtime metrics with base router config version
	s.runtimeMetrics = rmetric.NewRuntimeMetrics(
	s.logger,
	s.otlpMeterProvider,
	mappedMetricAttributes,

Engine observers are registered for each graph server using the shared s.engineStats:

cosmo/router/core/graph_server.go

Lines 659 to 680 in b9dd6b2

	// setupEngineStatistics creates the engine statistics for the server.
	// It creates the OTLP and Prometheus metrics for the engine statistics.
	func (s *graphServer) setupEngineStatistics(baseAttributes []attribute.KeyValue) (err error) {
	// We only include the base router config version in the attributes for the engine statistics.
	// Same approach is used for the runtime metrics.
	s.otlpEngineMetrics, err = rmetric.NewEngineMetrics(
	s.logger,
	baseAttributes,
	s.otlpMeterProvider,
	s.engineStats,
	&s.metricConfig.OpenTelemetry.EngineStats,
	s.metricConfig.OpenTelemetry.ResolverStats,
	)
	if err != nil {
	return err
	}
	s.prometheusEngineMetrics, err = rmetric.NewEngineMetrics(
	s.logger,
	baseAttributes,
	s.promMeterProvider,
	s.engineStats,

They are unregistered when the graph server is shut down:

cosmo/router/core/graph_server.go

Lines 2296 to 2304 in b9dd6b2

	if s.otlpEngineMetrics != nil {
	if err := s.otlpEngineMetrics.Shutdown(); err != nil {
	finalErr = errors.Join(finalErr, err)
	}
	}
	if s.prometheusEngineMetrics != nil {
	if err := s.prometheusEngineMetrics.Shutdown(); err != nil {
	finalErr = errors.Join(finalErr, err)

A plausible mechanism, which still needs a focused reproduction, is:

1. MessagesSent is cumulative and process-lived.
2. Another observer begins reporting the same shared engine statistics with a different config-version attribute set.
3. The changed attributes create a new OTLP metric series.
4. The first observation in that series contains the full pre-existing lifetime total.
5. Multiple observers or attribute sets coexist temporarily, duplicating gauges across series.

This would explain both the identical initial counter values and the simultaneous gauge multiplication, but the issue is intended to track the incorrect telemetry independently of whether this is the final mechanism.

Steps to reproduce / investigate

A deterministic minimal reproduction is still needed. The observed setup was:

1. Enable OTLP engine subscription metrics:

telemetry:
  metrics:
    otlp:
      enabled: true
      engine_stats:
        subscriptions: true

2. Start subscriptions and allow router.engine.messages.sent to accumulate a substantial lifetime value.
3. Keep the same router process running while its execution config/version changes through the normal control-plane lifecycle.
4. Query these metrics grouped by both process/pod identity and wg.router.config.version:
   • router.engine.messages.sent
   • router.engine.subscriptions
   • router.engine.triggers
5. Check whether a newly appearing config-version series begins with the existing cumulative MessagesSent value and whether gauges are concurrently present under multiple versions.

Other graph-server replacement or metrics-observer reinitialization paths may produce the same behavior and should also be tested.

Expected result

Metric-series lifecycle changes should not replay previously counted messages or inflate active engine gauges. Aggregating the router's emitted series should continue to represent actual fleet activity.

Actual result

New config-version series can begin with the full process-lifetime message count as fresh count volume. Active subscription and trigger gauges can also be represented more than once across concurrent series.

Environment information

• Linux container
• OTLP metrics exporter
• Execution config supplied by the control plane
• Same process and router binary across the observed metric-series transition
• Local router-config file watching/hot reload disabled

Router configuration

telemetry:
  metrics:
    otlp:
      enabled: true
      engine_stats:
        subscriptions: true

Router execution config

No graph-specific execution-config content appears necessary. The relevant observed condition is that new config-version metric series appeared while process-lifetime engine statistics already contained data.

Log output

Fetched router config
Router execution config has changed, hot reloading server
Fetched router config
Router execution config has changed, hot reloading server

Additional context

Potential fixes depend on the confirmed lifecycle mechanism. Options may include registering process-lifetime engine metrics once with stable attributes, excluding wg.router.config.version from lifetime engine counters, establishing a per-series baseline, or preventing multiple observers from concurrently reporting the same shared engine gauges.

[github-actions]

WunderGraph commits fully to Open Source and we want to make sure that we can help you as fast as possible.
The roadmap is driven by our customers and we have to prioritize issues that are important to them.
You can influence the priority by becoming a customer. Please contact us here.