MacOS VE - GPU passthrough block the host

[wpx]

some help please

I have a MacOS Monterey installed, it works fine, when I add the graphics card via pcie and start the virtual machine, the host crashes and I have to reboot... I have tried with a Windows 11 machine with the same result. ... I am new to virtualization but I have seen that it is highly recommended
..


My specs:
CPU: AMD Ryzen 9 5900X
MB: Gigabyte B450M DS3H
GPU: Strix RTX 580 8Gb



# dmesg | grep -e DMAR -e IOMMU
[ 0.644091] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.646598] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.646890] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

# dmesg | grep 'remapping'
[ 0.404995] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.646601] AMD-Vi: Interrupt remapping enabled

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:0c:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/15/devices/0000:09:00.0
/sys/kernel/iommu_groups/15/devices/0000:07:01.0
/sys/kernel/iommu_groups/15/devices/0000:07:04.0
/sys/kernel/iommu_groups/15/devices/0000:07:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.2
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.1
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.6
/sys/kernel/iommu_groups/13/devices/0000:00:18.4
/sys/kernel/iommu_groups/13/devices/0000:00:18.2
/sys/kernel/iommu_groups/13/devices/0000:00:18.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.7
/sys/kernel/iommu_groups/13/devices/0000:00:18.5
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:0d:00.4
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/18/devices/0000:0d:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
(GPU) /sys/kernel/iommu_groups/16/devices/0000:0b:00.0
(GPU audio) /sys/kernel/iommu_groups/16/devices/0000:0b:00.1
/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.3
/sys/kernel/iommu_groups/20/devices/0000:0d:00.3
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:0d:00.1
/sys/kernel/iommu_groups/9/devices/0000:00:07.1

# ls /sys/kernel/iommu_groups/16/devices/
0000:0b:00.0 0000:0b:00.1

# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67df,1002:aaf0 disable_vga=1

 

[leesteken]

Make sure not to use pcie_acs_override when checking the IOMMU groups. You probably have a single GPU that does not reset properly. Try vendor-reset; here is a guide.

 

[wpx]

I tried that but unfortunately this doesn't work for my... i think is for this reason


This is the error message if it helps...

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 'MacOS,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=4d722f25-af3a-464e-ae27-f31155079061' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/pve/vm-100-disk-0' -smp '12,sockets=3,cores=4,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -cpu 'Penryn,enforce,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,vendor=GenuineIntel' -m 32768 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=70e3c4e3-f3ec-4ed8-a42c-f8a1bb513bb5' -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:0b:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'usb-host,bus=xhci.0,hostbus=3,hostport=3,id=usb0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:62e53ca43bf5' -drive 'file=/var/lib/vz/template/iso/Monterey_2012.3.1_20v17.3.03.iso,if=none,id=drive-ide0,cache=unsafe,format=raw,aio=io_uring,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-virtio0,cache=unsafe,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=101' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'vmxnet3,mac=0E:E1:0C:75:65:20,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' -machine 'type=q35+pve0' -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -smbios 'type=2' -device 'usb-kbd,bus=ehci.0,port=2' -global 'nec-usb-xhci.msi=off' -cpu 'Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check'' failed: got timeout

 

[leesteken]

I tried that but unfortunately this doesn't work for my... i think is for this reason

View attachment 42475

Maybe it did not active? Are you using a kernel version 5.15 or later? Then you need to activate device_specific reset before starting the VM.
Try an Ubuntu VM, or just starting a VM with the Ubuntu Live installer DVD, that is a simple way to test if passthrough of the GPU working correctly.

This is the error message if it helps...

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 'MacOS,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=4d722f25-af3a-464e-ae27-f31155079061' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/pve/vm-100-disk-0' -smp '12,sockets=3,cores=4,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -cpu 'Penryn,enforce,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,vendor=GenuineIntel' -m 32768 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=70e3c4e3-f3ec-4ed8-a42c-f8a1bb513bb5' -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:0b:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'usb-host,bus=xhci.0,hostbus=3,hostport=3,id=usb0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:62e53ca43bf5' -drive 'file=/var/lib/vz/template/iso/Monterey_2012.3.1_20v17.3.03.iso,if=none,id=drive-ide0,cache=unsafe,format=raw,aio=io_uring,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-virtio0,cache=unsafe,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=101' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'vmxnet3,mac=0E:E1:0C:75:65:20,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' -machine 'type=q35+pve0' -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -smbios 'type=2' -device 'usb-kbd,bus=ehci.0,port=2' -global 'nec-usb-xhci.msi=off' -cpu 'Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check'' failed: got timeout

A timeout is usually cause by giving the VM too much memory. All the 32GB you give to the VM needs to be pinned into actual host memory because of PCI(e) passthrough. I don't know how much memory your system has.
Is there any (relevant) information in journalctl around the time of starting the VM? Can you attach the VM configuration file (/etc/pve/qemu-server/100.conf)?

 

[wpx]

kernel: (running kernel: 5.15.64-1-pve)

My vm config:

#cat /etc/pve/qemu-server/100.conf
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check
bios: ovmf
boot: order=ide0;virtio0;net0
cores: 4
cpu: Penryn
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,size=4M
hostpci0: 0000:0b:00.0,pcie=1,x-vga=1
ide0: local:iso/Monterey_2012.3.1_20v17.3.03.iso,cache=unsafe,size=14000M
machine: q35
memory: 16384
meta: creation-qemu=6.2.0,ctime=1666361072
name: MacOS
net0: vmxnet3=0E:E1:0C:75:65:20,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=4d722f25-af3a-464e-ae27-f31155079061
sockets: 3
usb0: host=3-3,usb3=1
virtio0: local-lvm:vm-100-disk-1,cache=unsafe,discard=on,size=800G
vmgenid: 70e3c4e3-f3ec-4ed8-a42c-f8a1bb513bb5

I would believe that the graphic works because when starting the host with a cable connected to the hdmi of the graphic card it shows the host console... or does this have nothing to do with it?

 

[wpx]

i have new advance... change the ram assigned from the vm to 16Gb and the VM start success but now i dont see the vm display in my screen

 

[leesteken]

A timeout is usually cause by giving the VM too much memory.

i have new advance... change the ram assigned from the vm to 16Gb and the VM start success but now i dont see the vm display in my screen

Good to know that you fixed the time-out.

Maybe it did not active? Are you using a kernel version 5.15 or later? Then you need to activate device_specific reset before starting the VM.

Does your GPU vendor-reset work? Did you enable device_specific (when using pve-kernel 5.15 or higher)? What does the command journalctl | grep AMD_POLARIS show?

 

[wpx]

# dmesg |grep vendedor

879] vendor_reset: loading out-of-tree module taints kernel.
[ 5.977513] vendor_reset_hook: installed

# journalctl | grep AMD_POLARIS
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2a3c
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2a44
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x28e8
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b58
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b4c
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x28dc
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2a40
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b58
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b58
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2950
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0

 

[wpx]

advance: i selected the gpu like primary gpu and set in graphics=none and mi screen stay in apple logo

for testing i created a new vm with windows 11 anda dded the gpu the result is the same... start and show the proxmox logo the boot and when the OS will start the screen (hdmi) its like disconnected but in console i see the OS start normally and work fine

if i remove the romfile from the pci gpu doesnt work even starting

 

[cwen]

fyi. PVE Intel GPU passthrough to MacOS monterey, no signal | Proxmox Support Forum

In win10, I can pass through the igpu output to the physical display, but macos has no signal output.