MacOS VE - GPU passthrough block the host

wpx

New Member
Oct 18, 2022
6
0
1
some help please

I have a MacOS Monterey installed, it works fine, when I add the graphics card via pcie and start the virtual machine, the host crashes and I have to reboot... I have tried with a Windows 11 machine with the same result. ... I am new to virtualization but I have seen that it is highly recommended
..


My specs:
CPU: AMD Ryzen 9 5900X
MB: Gigabyte B450M DS3H
GPU: Strix RTX 580 8Gb

1666377115734.png

# dmesg | grep -e DMAR -e IOMMU
[ 0.644091] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.646598] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.646890] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

# dmesg | grep 'remapping'
[ 0.404995] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.646601] AMD-Vi: Interrupt remapping enabled

# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:0c:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:05.0
/sys/kernel/iommu_groups/15/devices/0000:09:00.0
/sys/kernel/iommu_groups/15/devices/0000:07:01.0
/sys/kernel/iommu_groups/15/devices/0000:07:04.0
/sys/kernel/iommu_groups/15/devices/0000:07:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.2
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.1
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.3
/sys/kernel/iommu_groups/13/devices/0000:00:18.1
/sys/kernel/iommu_groups/13/devices/0000:00:18.6
/sys/kernel/iommu_groups/13/devices/0000:00:18.4
/sys/kernel/iommu_groups/13/devices/0000:00:18.2
/sys/kernel/iommu_groups/13/devices/0000:00:18.0
/sys/kernel/iommu_groups/13/devices/0000:00:18.7
/sys/kernel/iommu_groups/13/devices/0000:00:18.5
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:0d:00.4
/sys/kernel/iommu_groups/11/devices/0000:00:08.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/18/devices/0000:0d:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:07.0
(GPU) /sys/kernel/iommu_groups/16/devices/0000:0b:00.0
(GPU audio) /sys/kernel/iommu_groups/16/devices/0000:0b:00.1

/sys/kernel/iommu_groups/6/devices/0000:00:04.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/12/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:14.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.3
/sys/kernel/iommu_groups/20/devices/0000:0d:00.3
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:0d:00.1
/sys/kernel/iommu_groups/9/devices/0000:00:07.1

# ls /sys/kernel/iommu_groups/16/devices/
0000:0b:00.0 0000:0b:00.1

# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=1002:67df,1002:aaf0 disable_vga=1
 

leesteken

Famous Member
May 31, 2020
2,626
568
118
Make sure not to use pcie_acs_override when checking the IOMMU groups. You probably have a single GPU that does not reset properly. Try vendor-reset; here is a guide.
 

wpx

New Member
Oct 18, 2022
6
0
1
I tried that but unfortunately this doesn't work for my... i think is for this reason

1666405796979.png
This is the error message if it helps...

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 'MacOS,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=4d722f25-af3a-464e-ae27-f31155079061' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/pve/vm-100-disk-0' -smp '12,sockets=3,cores=4,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -cpu 'Penryn,enforce,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,vendor=GenuineIntel' -m 32768 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=70e3c4e3-f3ec-4ed8-a42c-f8a1bb513bb5' -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:0b:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'usb-host,bus=xhci.0,hostbus=3,hostport=3,id=usb0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:62e53ca43bf5' -drive 'file=/var/lib/vz/template/iso/Monterey_2012.3.1_20v17.3.03.iso,if=none,id=drive-ide0,cache=unsafe,format=raw,aio=io_uring,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-virtio0,cache=unsafe,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=101' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'vmxnet3,mac=0E:E1:0C:75:65:20,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' -machine 'type=q35+pve0' -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -smbios 'type=2' -device 'usb-kbd,bus=ehci.0,port=2' -global 'nec-usb-xhci.msi=off' -cpu 'Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check'' failed: got timeout
 
Last edited:

leesteken

Famous Member
May 31, 2020
2,626
568
118
I tried that but unfortunately this doesn't work for my... i think is for this reason

View attachment 42475
Maybe it did not active? Are you using a kernel version 5.15 or later? Then you need to activate device_specific reset before starting the VM.
Try an Ubuntu VM, or just starting a VM with the Ubuntu Live installer DVD, that is a simple way to test if passthrough of the GPU working correctly.
This is the error message if it helps...

TASK ERROR: start failed: command '/usr/bin/kvm -id 100 -name 'MacOS,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=4d722f25-af3a-464e-ae27-f31155079061' -drive 'if=pflash,unit=0,format=raw,readonly=on,file=/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,size=540672,file=/dev/pve/vm-100-disk-0' -smp '12,sockets=3,cores=4,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -cpu 'Penryn,enforce,kvm=off,+kvm_pv_eoi,+kvm_pv_unhalt,vendor=GenuineIntel' -m 32768 -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=70e3c4e3-f3ec-4ed8-a42c-f8a1bb513bb5' -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0000:0b:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -device 'usb-host,bus=xhci.0,hostbus=3,hostport=3,id=usb0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:62e53ca43bf5' -drive 'file=/var/lib/vz/template/iso/Monterey_2012.3.1_20v17.3.03.iso,if=none,id=drive-ide0,cache=unsafe,format=raw,aio=io_uring,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -drive 'file=/dev/pve/vm-100-disk-1,if=none,id=drive-virtio0,cache=unsafe,discard=on,format=raw,aio=io_uring,detect-zeroes=unmap' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=101' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'vmxnet3,mac=0E:E1:0C:75:65:20,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102' -machine 'type=q35+pve0' -device 'isa-applesmc,osk=ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc' -smbios 'type=2' -device 'usb-kbd,bus=ehci.0,port=2' -global 'nec-usb-xhci.msi=off' -cpu 'Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check'' failed: got timeout
A timeout is usually cause by giving the VM too much memory. All the 32GB you give to the VM needs to be pinned into actual host memory because of PCI(e) passthrough. I don't know how much memory your system has.
Is there any (relevant) information in journalctl around the time of starting the VM? Can you attach the VM configuration file (/etc/pve/qemu-server/100.conf)?
 

wpx

New Member
Oct 18, 2022
6
0
1
kernel: (running kernel: 5.15.64-1-pve)

My vm config:

#cat /etc/pve/qemu-server/100.conf
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -global nec-usb-xhci.msi=off -cpu Penryn,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+avx2,+aes,+fma,+fma4,+bmi1,+bmi2,+xsave,+xsaveopt,check
bios: ovmf
boot: order=ide0;virtio0;net0
cores: 4
cpu: Penryn
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,size=4M
hostpci0: 0000:0b:00.0,pcie=1,x-vga=1
ide0: local:iso/Monterey_2012.3.1_20v17.3.03.iso,cache=unsafe,size=14000M
machine: q35
memory: 16384
meta: creation-qemu=6.2.0,ctime=1666361072
name: MacOS
net0: vmxnet3=0E:E1:0C:75:65:20,bridge=vmbr0,firewall=1
numa: 0
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=4d722f25-af3a-464e-ae27-f31155079061
sockets: 3
usb0: host=3-3,usb3=1
virtio0: local-lvm:vm-100-disk-1,cache=unsafe,discard=on,size=800G
vmgenid: 70e3c4e3-f3ec-4ed8-a42c-f8a1bb513bb5

I would believe that the graphic works because when starting the host with a cable connected to the hdmi of the graphic card it shows the host console... or does this have nothing to do with it?
 
Last edited:

wpx

New Member
Oct 18, 2022
6
0
1
i have new advance... change the ram assigned from the vm to 16Gb and the VM start success but now i dont see the vm display in my screen
 

leesteken

Famous Member
May 31, 2020
2,626
568
118
A timeout is usually cause by giving the VM too much memory.
i have new advance... change the ram assigned from the vm to 16Gb and the VM start success but now i dont see the vm display in my screen
Good to know that you fixed the time-out.
Maybe it did not active? Are you using a kernel version 5.15 or later? Then you need to activate device_specific reset before starting the VM.
Does your GPU vendor-reset work? Did you enable device_specific (when using pve-kernel 5.15 or higher)? What does the command journalctl | grep AMD_POLARIS show?
 

wpx

New Member
Oct 18, 2022
6
0
1
# dmesg |grep vendedor

879] vendor_reset: loading out-of-tree module taints kernel.
[ 5.977513] vendor_reset_hook: installed

# journalctl | grep AMD_POLARIS
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2a3c
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:17 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2a44
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x28e8
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:19 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b58
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:33:34 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b4c
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 15:59:01 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x28dc
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:08 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2a40
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:10 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b58
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:11 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2b58
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:18 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: version 1.1
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing pre-reset
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing reset
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: CLOCK_CNTL: 0x0, PC: 0x2950
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: performing post-reset
Oct 25 16:03:26 pve-wpx kernel: vfio-pci 0000:0b:00.0: AMD_POLARIS10: reset result = 0
 

wpx

New Member
Oct 18, 2022
6
0
1
advance: i selected the gpu like primary gpu and set in graphics=none and mi screen stay in apple logo

for testing i created a new vm with windows 11 anda dded the gpu the result is the same... start and show the proxmox logo the boot and when the OS will start the screen (hdmi) its like disconnected but in console i see the OS start normally and work fine

if i remove the romfile from the pci gpu doesnt work even starting
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!