fix: report managed systemd unit health during reconcile - #578
Open
chubes4 wants to merge 1 commit into
Open
Conversation
Upgrade reconciled the unit file and said nothing about whether the
unit was running. On one host kimaki.service had been in ActiveState=
failed for four weeks while every upgrade printed 'Nothing changed —
everything was already up to date'. The live process was an unrelated
transient systemd-run unit that would not survive a reboot.
Add _report_systemd_unit_health, called on every reconcile pass
including the unchanged and dry-run paths — a long-dead unit whose file
is already correct is the quietest failure of all. Warnings are
collected in HEALTH_WARNINGS and resurfaced under 'Service health:' in
the final summary, separate from the file-level 'Updated:' list.
This reports only. 'NEVER restarts the unit — operator does that
explicitly' remains intact; an enabled-but-stopped unit is named along
with the exact start command rather than being started.
Guard the array append with 'declare -p' rather than the
"${arr+x}" idiom: an empty bash array reads as unset under the
latter, which would silently drop the first warning every time.
Fixes #576
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #576.
Problem
Upgrade reconciled the unit file and said nothing about whether the unit was running.
On one host
kimaki.servicehad been inActiveState=failedfor four weeks while every upgrade printed:The bot was actually served by an unrelated transient
systemd-rununit living in/run— invisible to the reconciler and unable to survive a reboot. A desired-state reconciler that reports only the half of the state it writes is not reporting state.Change
_report_systemd_unit_healthinbridges/_dispatch.sh, called on every reconcile pass — including the unchanged and dry-run paths. A long-dead unit whose file is already correct is the quietest failure of all, and the earlycmp -sreturn was hiding exactly that case.Warnings collect in
HEALTH_WARNINGSand resurface underService health:in the final summary, deliberately separate from the file-levelUpdated:list.Not an auto-restart
NEVER restarts the unit — operator does that explicitlyis a deliberate contract and stays intact. An enabled-but-stopped unit is named along with the exact start command. A disabled unit stays quiet, because that is an operator choice.One subtle bug worth flagging
The array-append guard uses
declare -p, not the${arr+x}idiom used elsewhere in this repo:An empty bash array reads as unset under
+x, so that guard would have silently dropped the first warning every time — the same class of silent failure this PR exists to fix. It caught itself in test. Note thatguidance/homeboy.shand others still use${UPDATED_ITEMS+x}; I left those alone to keep this PR scoped, but they have the same latent hole.Verification
bash tests/bridge-service-adapters.sh— PASS. Five new cases: active, failed, enabled-but-inactive, deliberately-disabled, and health-reported-when-file-unchanged, driven through a stubbedsystemctl.Also PASS:
bridge-render,agents-md-guidance,convergence-orchestrator.bash -nclean onupgrade.shandbridges/_dispatch.sh.tests/datamachine-worker.shfails identically on cleanmain(exit=1) and on this branch — pre-existing, baselined before assuming a regression.🤖 Authored by chubes-bot (Claude). Finalized outside Homeboy under explicit operator authorization: Cook could not admit the task on this host because its
opencoderoute is missingAI_PROVIDER_OPENAI_CODEX_*credentials. Bounded recovery was attempted first —--preview, nativehomeboy worktree create, backend rotation,homeboy upgrade0.360.4 → 0.367.9, andhomeboy daemon recover. Work was done in the linked isolated worktreewp-coding-agents@fix-issue-576-service-health.