v1.300070.0 leaks "dbus-daemon --session" procs with Prometheus/AppInsights SAP config, exhausts inotify ("too many open files") - fine on v1.300069.0

[allanalvaro]

Describe the bug

We run the agent across a fleet of EC2 Linux hosts and started getting "too many open files" out of the agent, plus a pile of dbus-daemon: Cannot initialize inotify in the journal. It traces back to leaked dbus-daemon processes.

On hosts where CloudWatch Application Insights (SAP monitoring) has added a prometheus scrape to the agent config, v1.300070.0 spawns a session bus (dbus-daemon ... --session, looks like a glib autolaunch) every time the agent starts or the config gets re-applied. The systemd unit ships with KillMode=process, so when the service stops those dbus procs don't get killed - they stay in the service cgroup and pile up. Each one holds an inotify instance + fds, so after a few days they blow past fs.inotify.max_user_instances (128 for root) and the agent can't add file watchers anymore.

v1.300069.0 doesn't do this. Same host, same config, same uptime, zero session dbus procs. It only started after the box auto-updated to v1.300070.0 (AWS-ConfigureAWSPackage pulling latest) at the end of June.

Steps to reproduce

• EC2 Linux, systemd + cgroup v2, agent running as root, installed via AWS-ConfigureAWSPackage (latest = v1.300070.0)
• Enable Application Insights SAP monitoring so it drops a prometheus config into amazon-cloudwatch-agent.d/ (file_sd scrape of a SAP_HOST_EXPORTER_* job)
• Let the config get re-applied periodically (State Manager etc) so the agent restarts
• watch 'pgrep -x dbus-daemon | wc -l' - the count climbs and never comes back down, and the journal fills with (dbus-daemon) remains running after unit stopped

What did you expect to see?

The agent shouldn't be autolaunching a session bus for prometheus/host collection, and whatever it does spawn should get cleaned up when the service stops instead of leaking.

What did you see instead?

dbus procs accumulating without limit. Log lines (hostnames/pids scrubbed):

systemd[1]: amazon-cloudwatch-agent.service: Unit process NNNN (dbus-daemon) remains running after unit stopped.
dbus-daemon[NNNN]: Cannot initialize inotify
start-amazon-cloudwatch-agent[NNNN]: level=ERROR source=file.go:243 msg="Error adding file watcher" discovery=file config=SAP_HOST_EXPORTER_xxxx err="too many open files"

The leaked procs look like:

dbus-daemon --syslog --fork --print-pid N --print-address N --session

For scale: on one box we went from 0 "remains running" events/week on v1.300069.0 to ~50k/day the week after it moved to v1.300070.0, nothing else changed. They're all in the amazon-cloudwatch-agent.service cgroup, which is why KillMode=process lets them survive the stop.

What version did you use?

v1.300070.0 (b1586, built with go1.26.4) is broken. v1.300069.0 (go1.26.3) is fine. Only 070 reproduces it for us.

What config did you use?

The prometheus config Application Insights generates (ids redacted):

{"logs":{"metrics_collected":{"prometheus":{
  "cluster_name":"applicationinsights-prometheus-cluster",
  "log_group_name":"Applicationinsights-Prometheus",
  "prometheus_config_path":"/opt/aws/amazon-cloudwatch-agent/applicationinsights_prometheus.yaml",
  "emf_processor":{"metric_declaration":[{"source_labels":["job"],
    "label_matcher":"^SAP_HOST_EXPORTER_xxxx$",
    "metric_selectors":["^sap_alerts_Availability$"]}]}}}}}

agent block uses run_as_user: root.

Environment

OS: Oracle Linux 9.7, kernel 5.15 UEK, cgroup v2, systemd 252. Stock amazon-cloudwatch-agent.service unit (KillMode=process, Restart=on-failure, RestartSec=60s).

Additional context

A couple of things I'm hoping you can help with:

• Any idea what changed between 069 and 070 on the prometheus/host path (or a bumped dependency) that makes it autolaunch a session bus now? That looks like the real trigger.
• Separate from that, could the packaged unit move to KillMode=mixed? That reaps the leftover dbus on stop and keeps this from snowballing regardless of what spawns it. The KillMode=process unit also comes up in Agent fails to start if log and trace configurations are omitted. #1320.

For anyone else who hits this: removing the AppInsights prometheus config stops new leaks, pkill -f 'dbus-daemon .*--session' reclaims the fds, and pinning the package to v1.300069.0 avoids it.