Skip to content

fix(vmm): reject undersized pci64 windows before gpu initialization - #1507

Open
Leechael wants to merge 1 commit into
nextfrom
fix/vmm-pci64-preflight
Open

Leechael wants to merge 1 commit into
nextfrom
fix/vmm-pci64-preflight

Conversation

@Leechael

Copy link
Copy Markdown
Collaborator

Problem

On usw5, eight B300 GPUs (10de:3182) expose 512 GiB + 64 MiB + 32 MiB of 64-bit MMIO BARs each: 4 TiB + 768 MiB in total. With qemu_pci_hole64_size = "4T", firmware expands the PCI64 window beyond the fixed window modeled by dstack-mr. KMS then rejects the ACPI digest after several minutes of boot, and repeated boots can interrupt GPU initialization/cleanup.

Fix

  • Check the resolved passthrough GPUs and NVSwitch devices before GPU sanitization or QEMU launch. Read only sysfs BAR resource rows 0–5 and sum 64-bit MMIO sizes, excluding ROM and SR-IOV VF resources.
  • Reject requirements larger than the configured PCI64 window (0 uses Q35's 32 GiB default). Return an actionable error, append the existing boot.error event and update boot_error, and clear the persisted started marker so automatic restart does not retry the same invalid configuration.
  • Reuse the check in the one-shot launch path.

This is intentionally a small capacity check, not a firmware allocation simulator: alignment gaps can still require a larger window. No guest/KMS/protobuf/Cloud API changes. Existing CreateVm cleanup on synchronous start failure is unchanged; retained VMs started through StartVm expose the error through VmInfo.events as before.

Verification

  • One compact inline regression test uses BAR rows captured from usw5 on 2026-10-30: eight B300s fail at 4 TiB and pass at 8 TiB. Also checks a derived 256 GiB BAR2 configuration, the default size, empty devices, and parsing/exclusion behavior.
  • Executed the exact helper and inline test in a temporary standalone Cargo harness with the repository's locked anyhow/fs-err versions: 1 test passed. Production helper Clippy checks passed in that harness. Surrounding GPU config types were represented by minimal matching structs; this is not a full VMM build.
  • Changed-file rustfmt checks (edition 2021) and git diff --check passed.
  • Full VMM tests were not run locally: the host is macOS with no running Linux container daemon, and the offline workspace test command is blocked by missing dcap-qvl-webpki 0.103.15+dcap.1. Linux integration/build checks are left to CI.

Signed-off-by: Leechael Yim <yanleech@gmail.com>
@Leechael
Leechael force-pushed the fix/vmm-pci64-preflight branch from 7484d3c to fb18c85 Compare October 30, 2026 08:33

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant