[1.7.0-beta2][Bug] Boot mount-storm knocks NetworkManager, logind, polkit, PID 1 & rauc off the system bus on ZimaCube Pro — network config & clean reboot lost (OTA too)

[jmakumbi]

What happened?

The whole system D-Bus control plane collapses ~28 seconds into every boot. A storm of systemd mount-unit signals (from Docker starting under ZimaOS's 3× media bind-mount layout) overruns the single-threaded dbus-broker, which then disconnects most root-owned core services from the system bus simultaneously. The processes keep running but lose their bus names and cannot re-acquire them until the next reboot.

This is much worse than "an update fails" — the box loses core functionality and remote recoverability:

Service (D-Bus name)	Process	Impact of losing its bus name
NetworkManager (org.freedesktop.NetworkManager)	alive	Networking can no longer be configured — web UI and nmcli both report "NetworkManager is not running" (it is). A bad network state can't be fixed without a reboot; risk of losing remote access entirely.
logind (org.freedesktop.login1)	alive	No clean reboot or shutdown — systemctl reboot → "Call to Reboot failed: Launch helper exited with unknown return code 1"; fans spin, machine never cycles. A physical power-cycle is the only way out.
polkit (org.freedesktop.PolicyKit1)	alive	Authorization framework down — PolicyKit-mediated privileged operations can no longer be authorized.
systemd / PID 1 (org.freedesktop.systemd1)	alive	The init manager's own bus interface is gone; anything driving systemd over D-Bus fails — including D-Bus activation of other services (below).
rauc (de.pengutronix.rauc)	alive	OTA updates fail — installer's RAUC.Info() → "The name de.pengutronix.rauc was not provided by any .service files".

Snapshot from an affected unit (uptime 1 h 25 m). Every listed process is confirmed alive via pgrep while owning no bus name:

names still OWNED on the system bus: 2  →  only org.freedesktop.DBus and org.freedesktop.Avahi
                                           (a healthy boot has ~20+)

org.freedesktop.NetworkManager   owned=false   NetworkManager   ALIVE (pid 1391)
org.freedesktop.login1           owned=false   systemd-logind   ALIVE (pid 1000)
org.freedesktop.PolicyKit1       owned=false   polkitd          ALIVE (pid 980)
org.freedesktop.systemd1         owned=false   systemd          ALIVE (pid 1)
de.pengutronix.rauc              owned=false   rauc             ALIVE (pid 983)

systemctl still appears to work (it uses systemd's private socket, not a bus name), which masks the outage and makes the box look "up" while its D-Bus control plane is dead.

The on-demand (D-Bus-activated) services are unreachable too

The services one might assume are "just idle" — systemd-hostnamed, systemd-timedated, systemd-localed, systemd-networkd, systemd-resolved, systemd-timesyncd, udisks2 — are D-Bus-activated: they start on demand when their name is called and idle-exit after ~30 s. During the wedge they cannot be started at all, because activation is routed through PID 1, which is itself off the bus. Every call fails with the same error as the failed reboot:

# timedatectl
Failed to query server: Launch helper exited with unknown return code 1
# hostnamectl
Failed to query system properties: Launch helper exited with unknown return code 1
# busctl call org.freedesktop.systemd1 … StartUnit systemd-hostnamed.service replace
Call failed: Launch helper exited with unknown return code 1

So the effective state is a near-total D-Bus control-plane collapse: only org.freedesktop.DBus (the broker) and org.freedesktop.Avahi keep a working name. Everything else is either severed but still running (NetworkManager, logind, polkit, systemd/PID 1, rauc) or impossible to activate (every on-demand service above). This is also why rauc reports "not provided by any .service files" — activation can't complete.

Severity — Critical for a headless NAS

Two of these failures remove any way to recover remotely: you can't reconfigure the network (NetworkManager) and you can't cleanly reboot (logind), so the only fix is physical access to power-cycle the box. For a device that's meant to run headless in a closet, that's the serious part. The blocked OTA is real but is the least of it.

Environment

• Device: ZimaCube Pro (12 logical cores) — the core count is the trigger (see below)
• Affected builds: 1.6.1, 1.6.2, 1.7.0-beta1, 1.7.0-beta2 (still unfixed; re-verified 2026-07-22)
• Not affected: my ZimaBoard / ZimaBoard2 / ZimaBlade (4-core) on the same builds
• Workload: 26 auto-starting Docker apps
• Component: dbus-broker (system bus) under the systemd mount-unit signal storm from Docker + ZimaOS's 3× media bind-mount layout

The exact knockout point (pinned with an active-polling probe)

I ran a probe that polls NameHasOwner on a fresh connection every 0.3 s (ground truth — the same view a new CLI client or rauc status gets), started before docker/rauc. It pins the failure to a single instant:

uptime 27.6s  overlay2 mounts = 0    rauc, NetworkManager, logind, systemd1 : all TRUE
uptime 27.9s  overlay2 mounts = 18   ← the 78 Docker overlay mounts begin landing in a rush
uptime 28.2s  overlay2 mounts = 78   all four STILL true
uptime 28.65s overlay2 mounts = 78   ALL FOUR = false  ← simultaneous mass-disconnect, never recovers

• The core names drop together, in a ≤0.3 s window, at system load ≈ 3.7 (low).
• It coincides exactly with the UnitNew/UnitRemoved burst peaking at ~830 signals/100 ms (~8,000/s) as the mounts are created.
• It is not CPU/load-driven — peak load (~19) happens ~75 s later and is irrelevant. I confirmed this directly (see "what doesn't work" below).

Root cause

1. dbus-broker is single-threaded and drains message queues serially.
2. ZimaOS bind-mounts the same storage at three paths, so every Docker overlay mount is tracked as three systemd mount units:
   • /media/SSD-Storage/docker/overlay2/<hash> (real)
   • /DATA/.media/SSD-Storage/docker/overlay2/<hash> (bind)
   • /var/lib/casaos_data/.media/SSD-Storage/docker/overlay2/<hash> (bind)
3. At boot, ~26 containers start in parallel and PID 1 broadcasts a burst of UnitNew/UnitRemoved signals (measured peak ~830/100 ms). The single-threaded broker can't drain the burst before its per-peer queues overflow, so it sheds all root-owned peers at once (it does not crash — NRestarts=0, same instance the whole boot).
4. The names are then unrecoverable while dbus-broker lives — only a reboot clears it. rauc is worse: it is registered both Type=dbus and D-Bus-activatable (SystemdService=rauc.service), so a later call to the unowned name triggers an activation that never completes, permanently wedging the name.

Why only the ZimaCube Pro: app count is similar to my 4-core boards. The difference is core count — 12 cores create the mounts far more in parallel, so the signal burst arrives faster than the single-threaded broker can drain. The 4-core boards create mounts more serially, stay under the drain rate, and never lose their services.

What generates the storm: across the 26 apps the mount churn splits evenly across the three bind-trees (measured 86/86/86) — a clean confirmation of the 3× multiplication. No single app is at fault; it's the aggregate of all apps' overlay mounts, each replicated ×3 by the stock layout.

Steps to reproduce

1. On a ZimaCube Pro (≥ ~8 logical cores), install ~15–24 apps so they auto-start at boot.
2. Reboot.
3. ~25–90 s in, watch the core bus names:

   for n in de.pengutronix.rauc org.freedesktop.NetworkManager org.freedesktop.login1 org.freedesktop.PolicyKit1; do
     printf "%s " "$n"; dbus-send --system --dest=org.freedesktop.DBus --print-reply \
       /org/freedesktop/DBus org.freedesktop.DBus.NameHasOwner string:$n | awk '/boolean/{print $2}'
   done
   

   They transition to false and never return, while pgrep shows the processes still alive.
4. Confirm the symptoms: nmcli general status → "NetworkManager is not running" / web UI can't change IP; systemctl reboot → "Launch helper exited with unknown return code 1", no reboot; timedatectl/hostnamectl → same "Launch helper" error; OTA install fails (RAUC.Info()).

See the trigger: dbus-monitor --system "type='signal',interface='org.freedesktop.systemd1.Manager',member='UnitRemoved'" — thousands of docker-*.mount events in the boot window.

Things that do NOT fix it (tested on the affected unit)

• CPU-priority protection of rauc.service (Nice=-15, CPUWeight=10000, OOMScoreAdjust=-900) — still dropped.
• CPU-priority protection of dbus-broker.service itself (CPUWeight=10000, Nice=-15) — no effect, mass-disconnect still fired at t≈27.8 s / load 2.2. This proves it's burst-volume/queue-overflow, not CPU scheduling.
• Delaying rauc start until the storm settles — worse: while rauc is nameless, zimaos-mark-good calls it over D-Bus and triggers the activation wedge, so rauc's later RequestName fails outright. Please don't delay a Type=dbus+activatable service.

Suggested fixes (first two attack the cause)

1. Cut the 3× mount-unit multiplication — mount the /DATA/.media and /var/lib/casaos_data/.media bind trees with slave/private propagation so Docker overlay mounts don't replicate into them (~3× fewer mount signals).
2. Serialize/throttle container auto-start so mount creation doesn't arrive as one wide parallel burst.
3. Raise dbus-broker's per-user/per-connection message-queue limits so a transient boot burst can't force it to shed peers.
4. Drop the SystemdService= activation coupling for rauc (it is already Type=dbus); the dual mechanism is what makes its name unrecoverable after a drop.
5. Make core daemons re-RequestName after a bus disconnect where possible (NetworkManager/logind normally do — verify they recover here).

Notes for triage

• dbus-broker does not crash (NRestarts=0), so crash-log hunting finds nothing. The tell is NameHasOwner=false for core services while their processes are alive.
• The journal is volatile (tmpfs) — capture across the boot with a persistent logger or the evidence is lost on reboot.
• This is the intersection of three separately-documented upstream issues: dbus-broker kicking core services off the bus (bus1/dbus-broker#361, #192), Docker+systemd mount-unit flooding (docker/for-linux#679), and heavy-container ZimaOS setups (#474).

I have per-boot captures (0.3 s ownership timelines, signal-rate CSVs, /proc curves) for both the baseline and the CPU-shield control boot, and can attach them on request.

Screenshots

No response