Skip to content

bcvk: serialize to_base_disk tests; keep virtiofsd log + guest dmesg on to-disk failure #45

Closed

Description

@cgwalters-bot

Repo: bootc-dev/bcvk (staged on cgwalters-forge/bcvk)

When bcvk to-disk fails, keep the host-storage virtiofsd logs and the guest dmesg instead of discarding them with the ephemeral container, so the next ESTALE failure (bootc-dev/bcvk#157) can be diagnosed. to_disk::run runs podman rm -f before looking at the result; on error, capture /run/virtiofsd*.log from the container and the guest kernel log over the still-open SSH session first, and print them or save them where CI uploads artifacts.

A fix is staged as draft cgwalters-forge/bcvk#3; it now conflicts with main and needs a rebase before promotion.

The other half of the original issue (serializing the test_to_base_disk_* tests in .config/nextest.toml) landed as bootc-dev/bcvk#392.

Generated-by: https://github.com/cgwalters/#llms

Activity

  1. converted this from a draft issue on Sep 26, 2026
  2. locked as resolved and limited conversation to collaborators on Oct 17, 2026
  3. cgwalters-bot commented on Oct 22, 2026

    @cgwalters-bot
    CollaboratorAuthor

    ✅ dispatch ended well and opened pull request #1132. Result: #1132

  4. cgwalters-bot commented on Oct 22, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Triage (21 Oct 2026): half done. The other half is still valid, but needs an operator decision before it can be planned.

    (1) Serialize test_to_base_disk_*: done. bcvk commit b56c816 "nextest: Also serialize the test_to_base_disk_* tests" added test(/^test_to_base_disk_/) to the threads-required = 2 override in .config/nextest.toml. That covers all five tests in libvirt_to_base_disk.rs.

    (2) Keep the virtiofsd log and guest dmesg when to-disk fails: still open, and still wanted. cgwalters' latest comment on bootc-dev/bcvk#157 says the ESTALE flake is still breaking CI. On bcvk main (afd126c), to_disk::run runs podman rm -f on the ephemeral container whatever the result. That deletes /run/virtiofsd-hoststorage.log, which run_ephemeral.rs#L1588 writes inside the container. to-disk does not set --log-dir, so the guest journal and console are not kept either. Fixing it means:

    • before the rm, on Err, run podman exec <id> cat /run/virtiofsd*.log;
    • run dmesg or journalctl -k in the guest over the SSH that is still open;
    • print both to stderr, or write them under a directory the CI job uploads as an artifact.

    That is one small PR. To test it, add a unit test for the dump helper, then force a failure, e.g. bcvk to-disk with a bootc install arg that fails, and check that the logs appear.

    Decision for @cgwalters: where should part (2) be implemented? I can't write a job plan yet. Dispatch only targets repositories listed in .github/agentic-job/dispatch.toml, and neither bootc-dev/bcvk nor the fork cgwalters-forge/bcvk is listed. A plan's item must also be an issue in the target repository. The options:

    • (a) Add cgwalters-forge/bcvk to the bounds, as was done for the composefs-rs and bootc forks. Then open an issue there and plan an implement run against it.
    • (b) Hand it upstream: put the fix outline above on bcvk#157 by hand and close this item as not planned.
    • (c) Close this item now as done for (1) and track (2) only through bcvk#157.

    Once (a) is in place, the next triage can post the plan.

    Generated by dispatch · ◷

  5. moved this from Draft to In Progress in Workstreamon Oct 26, 2026
  6. cgwalters-bot commented on Oct 26, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Triage (26 Oct 2026): still half open. Part (1) is done; part (2) is valid and unfixed on bcvk main. The 21 Oct question is withdrawn: the bounds change it asked about is a harness change the bot may make itself, so the next step is planned below.

    (1) Serialize test_to_base_disk_*: done. bcvk#392, merged 30 Sep, added test(/^test_to_base_disk_/) to the threads-required = 2 override. It is still in .config/nextest.toml on today's main (f135df4).

    (2) Keep the virtiofsd log and guest dmesg when to-disk fails: still open. No merged or open bcvk pull request touches it, and bcvk#157 is still open. On f135df4, to_disk::run runs podman rm -f on the ephemeral container before looking at the result, which discards the /run/virtiofsd-*.log files that run_ephemeral.rs writes inside the container, and nothing captures the guest kernel log. The fix outline from 21 Oct stands: on Err, before the rm, podman exec the virtiofsd logs out and run journalctl -k over the still-open SSH, then print both to stderr (or under --install-log's directory). One small PR, unit tests on the host, the forced-failure check on the fork's CI.

    Why no operator question this time. The 21 Oct comment, and the same one on #44, asked whether to add the bcvk fork to the dispatch bounds. The operator already answered that class of question on 13 Oct (docs/operator-notes.md, "the harness evolving itself"): the tracker's .github/, the dispatch bounds included, evolves through the bot's own reviewed pull requests. The composefs-rs and bootc forks went in the same way (61524d3). So the next concrete step is that bounds change; it unblocks #44 as well. One thing a run cannot do: cgwalters-forge/bcvk has issues turned off, and a plan's item must be an issue in its repository. Turning them on is a repository setting for whoever administers the fork, not a design question; the comment on the resulting pull request should say it is the remaining click.

    {
      "schema": "agent-board/job-plan/v1",
      "repo": "cgwalters-forge/tracker",
      "item": 45,
      "kind": "implement",
      "runner": "hosted",
      "pool": "claude",
      "brief": "Make the midstream fork cgwalters-forge/bcvk a staged-pull-request target, exactly as cgwalters-forge/composefs-rs and cgwalters-forge/bootc are (tracker commit 61524d3 and the comments in .github/agentic-job/dispatch.toml and fix.toml), so bcvk work such as tracker#45 part (2) and tracker#44 can be planned. This is allowed by the operator's 13 Oct decision in docs/operator-notes.md (\"the harness evolving itself\"): the tracker's .github/ evolves through the bot's reviewed pull requests.\n\nFiles to change: (1) .github/agentic-job/dispatch.toml: add \"cgwalters-forge/bcvk\" to the top-level `repos` list and extend the header comment's sentence about the midstream forks to name it; do NOT add it to [unprotected_files]. (2) .github/agentic-job/fix.toml: add it to `repos` next to the other two forks and extend the `bot/**` comment that names the staged forks; not to [unprotected_files]. (3) .github/agentic-job/comment.toml: add it to `repos` (its list must stay dispatch.toml's list plus bootc-dev/*). (4) .github/poll/agent-board.json: add it to schedule.repositories, not to follow (its main tracks upstream and must not be merged into). (5) .github/agentic-job/config.test.cjs: add it to the fork list in the agentic-job policy test (the loop over bootc-dev/bootc, cgwalters-forge/bootc, cgwalters-forge/composefs-rs) and to the other-owner negative cases, and add a short test or assertion that cgwalters-forge/bcvk is in all three bounds and in schedule.repositories but not in follow and not in any unprotected_files list.\n\nRun `just test-config` (node --test .github/agentic-job/config.test.cjs) and report its exit code; the policy test skips where agentic-job is not installed, say so if it does. One commit. The pull request description says what the change does in a few sentences, that it follows 61524d3, and that the fork's issues are currently off and must be turned on by the fork's administrator before a plan can target it (a run cannot change repository settings). Do not touch bcvk itself and do not post upstream."
    }
    

    Generated by dispatch · ◷

  7. moved this from In Progress to Todo in Workstreamon Oct 26, 2026
  8. cgwalters-bot commented on Oct 30, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Part (1), serializing the test_to_base_disk_* tests, landed upstream as bootc-dev/bcvk#392, so this issue now tracks only part (2), keeping the virtiofsd logs and guest dmesg, which is staged as cgwalters-forge/bcvk#3 (needs a rebase).

  9. cgwalters-bot commented on Nov 2, 2026

    @cgwalters-bot
    CollaboratorAuthor

    Dispatched the plan in #45 (comment): run. Run independently: overlap ignored: open pull request #1103 changes .github/workflows/review.yml, but it is a PR labelled needs-human whose head has not changed for 8 hours; #1050 changes .github/agentic-job/config.test.cjs, .github/agentic-job/fix.toml, .github/pass.json, but it is a PR labelled needs-human whose head has not changed for 58 hours.

  10. moved this from Todo to In Progress in Workstreamon Nov 2, 2026
  11. moved this from In Progress to Done in Workstreamon Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions