[Windows] CW Agent silent metric loss after Windows Update - no error logged, no recovery, requires reboot of instance

[rgoltz]

Describe the bug

After a Windows Update reconfigures installed MSI products (triggering Windows Modules Installer), PDH performance counter handles become invalid. The win_perf_counters plugin then fails on every scrape cycle with pdhErr=258 but:

1. The error is only logged at D! (Debug) level via log.Printf("D! metric init has error: %v", err)
2. The Gather() function returns nil (no error) to the caller
3. The OTel scrape controller in receiver/adapter/receiver.go treats this as a successful scrape with zero metrics
4. No metrics are sent to CloudWatch, but the agent continues running with no visible indication of failure

This results in silent metric loss in production. The only way to detect the problem is through external alarms on missing datapoints. The only recovery is to restart the agent process.

Steps to reproduce

1. Run CloudWatch Agent on Windows with win_perf_counters collecting LogicalDisk metrics
2. Allow Windows Update to run (specifically Windows Modules Installer / TrustedInstaller MSI reconfiguration)
3. After the update, the agent continues running but stops sending metrics

What did you expect to see?

• The agent should either recover by re-initializing PDH counter handles, or
• Log an E! (Error) level message indicating metrics collection has failed, or
• Return an error from Gather() so the OTel pipeline can report the failure

What did you see instead?

Agent process running, log file growing. Zero metrics reaching CloudWatch. CloudWatch alarm triggered after 70 minutes of missing data. Agent does not self-recover.

Service stop succeeded (SCM reported stopped), but agent processes remained running.

Service restart created a third process and all three coexisted, none delivering metrics. taskkill /F failed with Access Denied. Only an instance reboot resolved the situation, causing production downtime.

Agent log after PDH counter invalidation (debug level enabled):

# First failed scrape:
2026-07-11T18:50:42Z D! {"caller":"adapter/receiver.go:61","msg":"Begin scraping metrics with adapter","receiver":"win_perf_counters"}
2026-07-11T18:50:43Z D! metric init has error: (pdhErr=258) The system cannot find message text for message number 0x%1 in the message file for %2.

# Same error repeating indefinitely without escalation or recovery, only resolvable via instance reboot:
2026-07-14T11:41:42Z D! {"caller":"adapter/receiver.go:61","msg":"Begin scraping metrics with adapter","receiver":"win_perf_counters"}
2026-07-14T11:41:43Z D! metric init has error: (pdhErr=258) The system cannot find message text for message number 0x%1 in the message file for %2.

# Note: At default log level (info/warn), these lines would be completely invisible.
# No E! or W! is ever emitted. Gather() returns nil. Agent reports healthy.

Note: The logged value pdhErr=258 may represent a PDH status code that FormatMessage() cannot resolve (the error text "The system cannot find message text..." is the FormatMessage failure itself, not the PDH error description). The underlying PDH status is unclear from the log alone.

What version did you use?

Version: CWAgent/1.300069.0b1529 (go1.26.3; windows; amd64)

What config did you use?

Agent configured with win_perf_counters collecting LogicalDisk % Free Space metrics with dimensions [instance = C:], [InstanceId], [objectname = LogicalDisk]. Namespace: custom namespace. Collection interval: 60s.

Environment

OS: Windows Server 2016 (EC2, eu-central-1)
Agent running as: Windows Service (AmazonCloudWatchAgent)

Additional context

We traced the behavior to the following code pattern in plugins/inputs/win_perf_counters/win_perf_counters.go (Gather() method):

Note: Code references are based on the current main branch. The CWA fork may differ slightly from Telegraf upstream.

if !metric.initialized {
    if err := metric.init(); err != nil {
        log.Printf("D! metric init has error: %v", err)
        continue  // skips this metric silently
    }
}
// ...
return nil  // always returns nil, never an error

The item.init() method calls PdhOpenQuery and PdhAddEnglishCounter. After Windows Update invalidates the counter registry, these calls fail. The initialized field remains false, so init() is retried every cycle, but the error is never escalated.

In receiver/adapter/receiver.go, scrape() checks the return value of Gather():

if err := r.input.Input.Gather(r.accumulator); err != nil {
    // never reached because Gather() always returns nil
}
return r.accumulator.GetOtelMetrics(), nil // empty metrics, no error

Possible approaches (just ideas, not prescriptive):

• Escalate to W! or E! after N consecutive init() failures
• Return an error from Gather() when all metrics fail to initialize
• Attempt a full PDH handle re-initialization (close + reopen) periodically

The trigger in our case correlates with Windows Event ID 1035 (MsiInstaller reconfiguration of ~60 products including VC++ runtimes and the CloudWatch Agent MSI itself), caused by routine Windows Update activity (Windows Modules Installer / TrustedInstaller). The first missing CloudWatch datapoint aligns with this activity.

Related: #223 (Windows service two-process model)

Limitations of this report

• We did not run lodctr /R, winmgmt /resyncperf, or typeperf "\LogicalDisk(*)\% Free Space" before rebooting, so we cannot confirm whether PDH was broken system-wide or only within the agent process.
• No Microsoft-Windows-PerfCtrs events (ID 1008/1023) were found in the event log, but we did not specifically filter for this source at the time.