Skip to content

fix(tools): drop a failed version probe's stderr - #2498

Merged
mlorentedev merged 2 commits into
mainfrom
fix/version-probe-ignores-warnings
Oct 14, 2026
Merged

mlorentedev merged 2 commits into
mainfrom
fix/version-probe-ignores-warnings

Conversation

@mlorentedev

@mlorentedev mlorentedev commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

The from-zero run behind BUG-138 reported ansible-lint: reports 2.21.5 after install, below the pin 26.9.0. ansible-lint had crashed with exit 3, and 2.21.5 came from ansible-core's UNSUPPORTED warning on stderr. The probe read stdout then stderr and kept the output of a failed run.

  • New tools.ProbeRunner: stdout then stderr, as before, but a failed run gives its stdout alone. It is the default probe of ProbeVersionArgs, so dotf tools install's three probe sites use it, and dotf tools version too.
  • tools.ExecRunner is unchanged. herdr formats a failed command's stderr into its errors, and that stays.
  • Doctor's semverOf (copilot, opencode, pi) went through CommandOutput, a CombinedOutput. That stream was also interleaved, against the stdout-first rule. It now goes through a new System.VersionOutput seam, tools.ProbeRunner in production. A System without the seam uses its CommandOutput, so a test never runs the real binary.
  • The documented cases still hold: a version on stdout before an unrelated failure, a warning on stderr before a stdout banner, and a version only on stderr from a run that exits 0.

Closes #2494

Not in this PR: MiseSync.runsAtPin probes through the merged mise runner, the same class with a boolean result. Filed as BUG-142 (#2496), since MiseSync needs its own probe seam.

Verification

$ go build ./... && go vet ./... && GOOS=windows go vet ./... && go test ./...
(exit 0; every package ok)
$ golangci-lint cache clean && golangci-lint run ./...
0 issues.

New cases in TestProbeRunner_ReadsStdoutBeforeStderr run real sh -c scripts: echo 'UserWarning: UNSUPPORTED: ansible-core 2.21.5 ...' >&2; exit 3 gives "", and echo 'tool 2.3.4'; echo 'error: ...' >&2; exit 1 gives 2.3.4.

Mutations, each caught:

  • ProbeRunner keeps stderr on failure: TestProbeRunner_ReadsStdoutBeforeStderr and TestRealSystem_VersionOutputDropsAFailedRunsStderr fail.
  • toolsVersionRunner back to tools.ExecRunner: TestToolsVersion_DropsAFailedRunsStderr fails.
  • No fallback for a nil VersionOutput: TestSemverOf_WithoutTheSeamUsesCommandOutput fails.

Production diff: 49 lines.

Knowledge

A tool that crashes leaves on stderr the warnings of whatever it loaded.
ansible-lint exited 3 with only ansible-core's UNSUPPORTED warning to
read, and the probe reported it installed at ansible-core's 2.21.5.

ProbeRunner keeps stdout then stderr, but gives a failed run's stdout
alone. It is the default probe for dotf tools and, through a new
System.VersionOutput seam, for doctor's semverOf, which read a merged
CombinedOutput stream.

Closes #2494
@mlorentedev mlorentedev added the bug Something isn't working label Oct 14, 2026
@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: mlorentedev/dotfiles/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8523780d-4172-46e7-8159-41f5583c3bc3
📥 Commits

Reviewing files that changed from the base of the PR and between 3576de3 and 049206b.

📒 Files selected for processing: 9
  • cli/internal/cmd/tools.go
  • cli/internal/cmd/tools_version_test.go
  • cli/internal/doctor/checks_deploy.go
  • cli/internal/doctor/system.go
  • cli/internal/doctor/testhelpers_test.go
  • cli/internal/doctor/version_output_test.go
  • cli/internal/tools/install.go
  • cli/internal/tools/version.go
  • cli/internal/tools/version_test.go
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Comment thread cli/internal/doctor/checks_deploy.go Outdated
out, err := sys.CommandOutput(n, args...)
return []byte(out), err
})
v := tools.ProbeVersion(name, sys.VersionOutput)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possible Issue

semverOf now passes sys.VersionOutput to tools.ProbeVersion, which substitutes tools.ProbeRunner when the runner is nil — so a System built without VersionOutput bypasses the CommandOutput fake and shells out to the real binary. In this PR only realSystem() and newSys populate the field; any other doctor test helper or inline System literal that fakes CommandOutput with a version banner but omits VersionOutput will read the machine's actual installed version (test passes with the wrong value) or get "" → "unknown" instead of the injected banner. THEORETICAL: the diff shows no such second constructor, so this cannot be confirmed from the hunks alone — grep for System{ literals outside realSystem/newSys before merging.

@github-actions

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

2494 - Partially compliant

Compliant requirements:

  • On a failed run the probe drops stderr, so a crash's warning semver never supplies a version (the BUG-141 scenario, ansible-lint exit 3 with ansible-core 2.21.5 on stderr, reports no version).
  • Tests added: TestProbeRunner_ReadsStdoutBeforeStderr case "a crash that names another package's version", plus TestToolsVersion_DropsAFailedRunsStderr and TestRealSystem_VersionOutputDropsAFailedRunsStderr.
  • The wiring covers dotf tools install (nil Probe now defaults to ProbeRunner), dotf tools version (toolsVersionRunner), and doctor's semverOf (System.VersionOutput).

Non-compliant requirements:

  • Option A ("a command that exits non-zero reports no version") is not implemented: ProbeRunner still returns a failed run's stdout, so a tool that crashes after printing a version on stdout still reports it.

Requires further human verification:

  • Whether keeping a failed run's stdout is the accepted reading of the "Or" in the ticket, or whether the strict option A was wanted (the PR deliberately preserves the "prints the version, then complains" case).

2496 - Partially compliant

Compliant requirements:

  • (empty)

Non-compliant requirements:

  • MiseSync gains no Probe seam; runsAtPin still probes through the merged runner.
  • No test for a failing probe whose stderr holds a pin-satisfying version.

Requires further human verification:

  • None: the PR body explicitly declares BUG-142 out of scope and files it, so the non-compliance is a deliberate deferral, not an omission.
⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪
🧪 PR contains tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Harness compliance
Definition of Done §2: the PR must name knowledge in a ## Knowledge section, one line each for Lesson, ADR and Runbook, or none: <reason>. The submitted PR description has Summary and Verification sections only; no knowledge disposition of any kind appears. Where the BUG-141 probe lesson belongs (e.g. docs/lessons.md) is left unstated.

Harness compliance
Spec-Driven Development: the change touches public contracts (tools.ProbeRunner is a new exported function, System.VersionOutput a new exported field) but the diff adds no specs/<id>/ folder, and no skip-sdd label with written rationale is visible in the PR metadata provided for review. If the PR does carry a skip-sdd label with rationale outside that metadata, this item passes; as submitted here it cannot be confirmed. Confidence limited by not seeing PR labels.


⚙️ Agent run details
  • Model: openai/mimo-v2.6-flash
  • Tokens: 18,896 in / 7,809 out / 26,705 total
  • Time cost: 144.3s
  • AI calls: 1

A nil VersionOutput made ProbeVersion fall back to the real ProbeRunner,
so a test System that faked CommandOutput would run the machine's
binary. Nil now means CommandOutput.
@mlorentedev

Copy link
Copy Markdown
Owner Author

Review triage

Reviewers: PR-Agent reviewed aa0908ba. CodeRabbit posted only its review-limit notice, so it did not review.

Finding Disposition
checks_deploy.go:936: a System without VersionOutput makes ProbeVersion run the real binary Applied in 049206b5: a nil VersionOutput now means CommandOutput, so a test System stays on its fake. TestSemverOf_WithoutTheSeamUsesCommandOutput covers it, and removing the fallback fails it.
Option A not implemented: a failed run's stdout still counts Declined: the ticket offered two options joined by "Or". This PR takes the stdout-only reading, because ProbeVersion documents a real case it must keep: a tool that prints its version and then fails for an unrelated reason. Both cases are pinned in TestProbeRunner_ReadsStdoutBeforeStderr.
MiseSync gains no probe seam Declined here: that is BUG-142 (#2496), named in the body. Its fix is ready and lands right after this PR, stacked on it.
No ## Knowledge section Declined: the body has one (lesson 429 already covers it). The reviewer saw a truncated body.
SDD gate: new exported ProbeRunner and System.VersionOutput Declined: both live under cli/internal/, which no other module can import, so they are no public contract. The production diff is 49 lines, under the 50-line gate, and spec-gate passes.

049206b5 was pushed after the review and has not been re-reviewed.

@mlorentedev
mlorentedev merged commit 2fce9f6 into main Oct 14, 2026
25 of 28 checks passed
@mlorentedev
mlorentedev deleted the fix/version-probe-ignores-warnings branch October 14, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working Review effort 2/5

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG-141: the version probe reads a semver out of a crashing tool's warning

1 participant