Skip to content

fix: report managed systemd unit health during reconcile - #578

Open
chubes4 wants to merge 1 commit into
mainfrom
fix/issue-576-service-health
Open

fix: report managed systemd unit health during reconcile#578
chubes4 wants to merge 1 commit into
mainfrom
fix/issue-576-service-health

Conversation

@chubes4

@chubes4 chubes4 commented Sep 4, 2026

Copy link
Copy Markdown
Member

Fixes #576.

Problem

Upgrade reconciled the unit file and said nothing about whether the unit was running.

On one host kimaki.service had been in ActiveState=failed for four weeks while every upgrade printed:

Nothing changed — everything was already up to date.

The bot was actually served by an unrelated transient systemd-run unit living in /run — invisible to the reconciler and unable to survive a reboot. A desired-state reconciler that reports only the half of the state it writes is not reporting state.

Change

_report_systemd_unit_health in bridges/_dispatch.sh, called on every reconcile pass — including the unchanged and dry-run paths. A long-dead unit whose file is already correct is the quietest failure of all, and the early cmp -s return was hiding exactly that case.

Warnings collect in HEALTH_WARNINGS and resurface under Service health: in the final summary, deliberately separate from the file-level Updated: list.

Not an auto-restart

NEVER restarts the unit — operator does that explicitly is a deliberate contract and stays intact. An enabled-but-stopped unit is named along with the exact start command. A disabled unit stays quiet, because that is an operator choice.

One subtle bug worth flagging

The array-append guard uses declare -p, not the ${arr+x} idiom used elsewhere in this repo:

arr=(); [ -n "${arr+x}" ]   # reads as UNSET

An empty bash array reads as unset under +x, so that guard would have silently dropped the first warning every time — the same class of silent failure this PR exists to fix. It caught itself in test. Note that guidance/homeboy.sh and others still use ${UPDATED_ITEMS+x}; I left those alone to keep this PR scoped, but they have the same latent hole.

Verification

bash tests/bridge-service-adapters.sh — PASS. Five new cases: active, failed, enabled-but-inactive, deliberately-disabled, and health-reported-when-file-unchanged, driven through a stubbed systemctl.

Also PASS: bridge-render, agents-md-guidance, convergence-orchestrator. bash -n clean on upgrade.sh and bridges/_dispatch.sh.

tests/datamachine-worker.sh fails identically on clean main (exit=1) and on this branch — pre-existing, baselined before assuming a regression.


🤖 Authored by chubes-bot (Claude). Finalized outside Homeboy under explicit operator authorization: Cook could not admit the task on this host because its opencode route is missing AI_PROVIDER_OPENAI_CODEX_* credentials. Bounded recovery was attempted first — --preview, native homeboy worktree create, backend rotation, homeboy upgrade 0.360.4 → 0.367.9, and homeboy daemon recover. Work was done in the linked isolated worktree wp-coding-agents@fix-issue-576-service-health.

Upgrade reconciled the unit file and said nothing about whether the
unit was running. On one host kimaki.service had been in ActiveState=
failed for four weeks while every upgrade printed 'Nothing changed —
everything was already up to date'. The live process was an unrelated
transient systemd-run unit that would not survive a reboot.

Add _report_systemd_unit_health, called on every reconcile pass
including the unchanged and dry-run paths — a long-dead unit whose file
is already correct is the quietest failure of all. Warnings are
collected in HEALTH_WARNINGS and resurfaced under 'Service health:' in
the final summary, separate from the file-level 'Updated:' list.

This reports only. 'NEVER restarts the unit — operator does that
explicitly' remains intact; an enabled-but-stopped unit is named along
with the exact start command rather than being started.

Guard the array append with 'declare -p' rather than the
"${arr+x}" idiom: an empty bash array reads as unset under the
latter, which would silently drop the first warning every time.

Fixes #576
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Service reconcile reports success for a unit that has been failed for weeks

1 participant