AMD GPU Passthrough. Device is already attached.

[noidedxyz]

I'm on Arch Linux and I am running a nVidia 3060 and have a 5600xt as well but am only using the nvidia card on my main os. (AMD 2600x CPU, btw)

     boot.sh
     macOS Sonoma

     Built with ULTMOS v0.12.1
     Using Haswell-noTSX CPU model
     Superuser privileges enabled
     Passthrough enabled
     Discord RPC enabled
  ──────────────────────────────────────────────

QEMU 9.0.1 monitor - type 'help' for more information
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.erms [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.erms [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.erms [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.erms [bit 9]
qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.invpcid [bit 10]
(qemu) qemu-system-x86_64: -device vfio-pci,host=2a:00.0,multifunction=on,bus=pcie.0: vfio 0000:2a:00.0: device is already attached

I'm not entirely sure what to do; I pass all the passthrough checks and assumed it should work. I am using systemd-boot/dracut if that matters. The mac emulation appears to work fine without gpu passthrough but feels kind of unusable. I have been trying this for 2 days now and I am unsure now what to do. It acts like the amdgpu is still attached when it isn't even in use. Please point me in the right direction for help. I am completely lost here.

[noidedxyz]

Here's my boot.sh if it helps

#!/usr/bin/env bash
# shellcheck disable=SC2054

#
#   APC-RUN_26-06-2024_15-40-49
#
#   THIS FILE WAS GENERATED USING AUTOPILOT.
#
#   To boot this script, run the following command:
#   $ ./boot.sh
#

#
#	boot.sh
#	Created by Coopydood as part of the ultimate-macOS-KVM project.
#
#	Profile: https://github.com/Coopydood
#	Repo: https://github.com/Coopydood/ultimate-macOS-KVM
#
#	Adapted from OSX-KVM among others.
#	Greetz to TheNickDude, Dortania, khoalia, foxlet, and other contributors :]
#


ID="macOS"
NAME="macOS 14"
FILE="boot.sh"

ULTMOS=0.12.1
IGNORE_FILE=0
REQUIRES_SUDO=1
VFIO_PTA=1
VFIO_DEVICES=4
GEN_EPOCH=1719434449
FEATURE_LEVEL=7
VERBOSE=1
DISCORD_RPC=1
DISCORD_RPC_IMG="default"

SCREEN_RES="1280x720"

ALLOCATED_RAM="8G"
CPU_SOCKETS="1"
CPU_CORES="2"
CPU_THREADS="4"
CPU_MODEL="Haswell-noTSX"
CPU_FEATURE_ARGS="+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check"

REPO_PATH="/run/media/noided/NVMe Storage/VirtualMac"
OVMF_DIR="/run/media/noided/NVMe Storage/VirtualMac/ovmf"

VFIO_ID_0="$USR_VFIO_ID_0"
VFIO_ID_1="$USR_VFIO_ID_1"
VFIO_ROM="$USR_VFIO_ROM"

USB_DEVICES="$USR_USB_DEVICES"

NETWORK_DEVICE="vmxnet3"
MAC_ADDRESS="00:16:cb:00:21:09"

OS_ID="Sonoma"

HDD_PATH="$REPO_PATH/HDD.qcow2"
DISK_TYPE="HDD"

#   You should not have to touch anything below this line, especially if you
#   don't really know what you're doing. It'll probably break something.

args=(
-global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off
-enable-kvm -m "$ALLOCATED_RAM" -cpu "$CPU_MODEL",kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,"$CPU_FEATURE_ARGS"
-machine q35
-boot menu=on,splash-time=5
##-device usb-ehci,id=ehci
#-device qemu-xhci,id=xhci
-usb -device usb-kbd -device usb-tablet 
#USB_DEV_BEGIN
#USB_DEV_END
-smp "$CPU_THREADS",cores="$CPU_CORES",sockets="$CPU_SOCKETS"
-device pcie-root-port,bus=pcie.0,slot=1,x-speed=16,x-width=32
#VFIO_DEV_BEGIN
-device vfio-pci,host="2a:00.0",multifunction=on,bus=pcie.0
-device vfio-pci,host="2a:00.1",bus=pcie.0
-device vfio-pci,host="2a:00.0",multifunction=on,bus=pcie.0
-device vfio-pci,host="2a:00.0",romfile="./roms/",multifunction=on,bus=pcie.0
-device vfio-pci,host="2a:00.1",bus=pcie.0
#VFIO_DEV_END
-device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc"
-drive if=pflash,format=raw,readonly=on,file="$OVMF_DIR/OVMF_CODE.fd"
-drive if=pflash,format=raw,file="$OVMF_DIR/OVMF_VARS_PT.fd"
-smbios type=2
-device ich9-intel-hda -device hda-duplex
-device ich9-ahci,id=sata
-drive id=OpenCore,if=none,format=qcow2,file="$REPO_PATH/boot/OpenCore.qcow2"
-drive id=HDD,if=none,file="$HDD_PATH",format=qcow2
-device ide-hd,bus=sata.2,drive=OpenCore,bootindex=1
-device ide-hd,bus=sata.3,drive=HDD,rotation_rate=7200

############## REMOVE THESE LINES AFTER MACOS INSTALLATION ###############
#-drive id=BaseSystem,if=none,file="$REPO_PATH/BaseSystem.img",format=raw
#-device ide-hd,bus=sata.4,drive=BaseSystem
##########################################################################

-netdev user,id=net0 -device "$NETWORK_DEVICE",netdev=net0,id=net0,mac="$MAC_ADDRESS"
#-device qxl-vga,vgamem_mb=128,vram_size_mb=128   # DISABLED BY VFIO-PCI PASSTHROUGH ASSISTANT
-monitor stdio
-display none
-vga none

################ UNCOMMENT IF YOU WANT TO USE VNC MONITOR ################
#-vnc 0.0.0.0:1,password=on -k en-us
##########################################################################

)

while getopts d: flag
do
    case "${flag}" in
        d) DISCORD_RPC=${OPTARG};;
    esac
done

if [ $VERBOSE = 1 ]
then
echo
echo \ \ \──────────────────────────────────────────────
echo \ \ \ \ \ $FILE
echo \ \ \ \ \ $ID $OS_ID
echo
echo \ \ \ \ \ Built with ULTMOS v$ULTMOS
echo \ \ \ \ \ Using $CPU_MODEL CPU model
if [ $REQUIRES_SUDO = 1 ]
then
echo \ \ \ \ \ Superuser privileges enabled
fi
if [ $VFIO_PTA = 1 ]
then
echo \ \ \ \ \ Passthrough enabled
else
echo \ \ \ \ \ Passthrough disabled
fi
if [ $DISCORD_RPC = 1 ]
then
echo \ \ \ \ \ Discord RPC enabled
else
echo \ \ \ \ \ Discord RPC disabled
fi
echo \ \ \──────────────────────────────────────────────
echo
fi

if [ $DISCORD_RPC = 1 ]
then
"$REPO_PATH/scripts/drpc.py" --os "$OS_ID" --pt $VFIO_DEVICES --wd "$REPO_PATH" --show "$DISCORD_RPC_IMG" &
fi

qemu-system-x86_64 "${args[@]}"

if [ $DISCORD_RPC = 1 ]
then
pkill -f drpc.py
fi

[Coopydood]

Hi there,

Don’t worry, I’ll help you get it working. I’m happy to help here, or for likely faster replies I’m also on discord.

Either way, please start by running the sysprofiler for me, accessed by: Main menu > Compatibility checks > Run system profiler

thanks :)

[noidedxyz]

Here you go, thanks.

ULTMOS SYSTEM REPORT 26-06-2024 18:19:26
───────────────────────────────────────────────────────────────────
This report was generated by user request, and includes basic
information about your system hardware, as well as your current
ULTMOS software environment.

This information may help project developers
in assisting you in any issues you might have.
    
   Name       : System Profile Tool
   File       : sysprofiler.py
   Identifier : SPT
   Version    : 1.0
   Vendor     : Coopydood
    
   Date       : 26/06/2024
   Time       : 18:19:28
    

   ULTMOS
   ────────────────────────────────────────────────────────
   Version    : 0.12.1
   Branch     : main
    

   OPERATING SYSTEM
   ────────────────────────────────────────────────────────
   OS         : Linux
   Distro     : Arch Linux
   Release    : rolling
   Kernel     : 6.9.6-arch1-1
   BootCMD    : initrd=\f0794a2161144e6bb883900d074c3daf\6.9.6-arch1-1\initrd nvme_load=YES nowatchdog rw rootflags=subvol=/@ root=UUID=38b5f14e-2fa7-4e60-9900-d5e8c5c21e47 resume=UUID=5fe7f975-e482-4e4c-a421-7896d3805fd7 nvidia_drm.modeset=1 nvidia-drm.fbdev=1 amd_iommu=1 iommu=pt vfio-pci.ids=1002:731f,1002:ab38 systemd.machine_id=f0794a2161144e6bb883900d074c3daf
   VM         : NO
    

   PROCESSOR
   ────────────────────────────────────────────────────────
   Model      : AMD Ryzen 5 2600X Six-Core Processor
   Physical   : 6
   Logical    : 12
   Arch       : x86_64
    

   MEMORY
   ────────────────────────────────────────────────────────
   Total      : 15.55 GB
   Used       : 4.08 GB
   Available  : 10.92 GB
   Free       : 8.55 GB
    

   CONFIGURATION
   ────────────────────────────────────────────────────────
   BootMode   : UEFI
   IOMMU      : Available
   Libvirtd   : Enabled
    
   QEMU       : Installed
   Libvirt    : Installed
   Virsh      : Installed
   VirtMan    : Installed
   Pypresence : Included
   NBD        : Installed
    

   VFIO-PCI
   ────────────────────────────────────────────────────────
   Kernel     : Configured
   Stubbed    : 2
   Device0    : 2a:00.0 Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] 
   Device1    : 2a:00.1 Navi 10 HDMI Audio 
    

   INTEGRITY
   ────────────────────────────────────────────────────────
   Core       : PASS
   ScriptStr  : PASS
   OVMFStr    : PASS
   BaseFiles  : PASS
   InclBin    : PASS
   DeltaData  : PASS
    

   AUTOPILOT
   ────────────────────────────────────────────────────────
   FeatureLvl : 8
   UserBlobs  : YES (17 total)
                ├ USR_HDD_SIZE.apb
                ├ USR_MAC_ADDRESS.apb
                ├ USR_NETWORK_DEVICE.apb
                ├ USR_CPU_CORES.apb
                ├ USR_TARGET_OS.apb
                ├ USR_CFG.apb
                ├ USR_VFIO_DEVICES.apb
                ├ USR_CPU_THREADS.apb
                ├ USR_CPU_FEATURE_ARGS.apb
                ├ USR_HDD_TYPE.apb
                ├ USR_SCREEN_RES.apb
                ├ USR_CPU_MODEL.apb
                ├ USR_ALLOCATED_RAM.apb
                ├ USR_TARGET_OS_NAME.apb
                ├ USR_CREATE_XML.apb
                ├ USR_BOOT_FILE.apb
                └ USR_HDD_PATH.apb
    
   StaleBlobs : YES (16 total)
   LiveBlobs  : NO
    
   OCInPlace  : YES
⚠ OCModded   : YES
   OCSize     : 27.00 MB
⚠ OCHashMD5  : d0818142b34274e67ffdd3541e3fd0c8
    
   APFileName : boot.sh
   APFilePath : /run/media/noided/NVMe Storage/VirtualMac/boot.sh
    

   GENERATED DATA // BOOT.SH
   ────────────────────────────────────────────────────────
   Name       : boot.sh
   Path       : /run/media/noided/NVMe Storage/VirtualMac/boot.sh
    
   ULTMOS     : 0.12.1
   GenTime    : 1719434449
    
   OS         : macOS Sonoma
   Version    : 14
    
   DiskPath   : /run/media/noided/NVMe Storage/VirtualMac/HDD.qcow2
   DiskUsed   : 28.81 GB
   DiskSize   : 256 GB
   DiskType   : HDD
   DiskIsReal : NO
    
   RecImgPath : /run/media/noided/NVMe Storage/VirtualMac/BaseSystem.img
   RecImgSize : 2.99 GB
   RecImgHash : 293d705c43a555f5b47675f0ddff04e2
   RecImgFrom : Downloaded with APC
    

    

   WARNINGS (2)
   ────────────────────────────────────────────────────────
   OpenCore image is very likely to have been modified
   by the user, integrity can't be verified

   OpenCore image MD5 hash does not match any stock
   OC images supplied with the project

[noidedxyz]

Yeah I don't know why it added it 3 times but I removed the extra lines. The MD5 hash is indeed due to the pikera mod. So now the "device is already attached" issue is gone but nothing really appears to open. Just seems like qemu is running. I'm new to qemu. Is there another step I am missing to display the OS running? Thanks again!

[Coopydood]

The only other step is to make sure a display is connected but again I assume you’ve done that!

One thing I would highly recommend with pass through is to use virt manager instead of the standard script. For reasons I don’t know, pass through works better using virt-manager, whereas the opposite is true when not using pass through.

If you haven’t already, convert your script: Main menu > Extras > Convert to XML

Import it, and then try running it using virtual machine manager. It will appear as “macOS Sonoma (ULTMOS)”

[Coopydood]

Oh, and if you experience a “hang” during the macOS boot process, it may be a red herring. Please allow up to 15 minutes before deeming it dead. If it does boot after a long wait, it’s simply “the verbose bug”. You can fix this by removing the verbose boot argument from the macOS boot args.

[noidedxyz]

I have solved this, my own stupid mistake. It has been working I just didn't realize I needed to take my hdmi cable to the card itself to see whats going on. That's my bad. I appreciate the help.

[Coopydood]

No worries, believe me stupid mistakes are all part of getting pass through to work 😂

glad you got it sorted. If you need anything else don’t hesitate to reach out. :)