Host Crash and Reboot Everytime Running Multiple VM with GPU and iGPU Passthrough

X

x01

New Member
Feb 9, 2021
4
0
1
34
Long story short, I successfully installed windows 10 with 6900xt gpu passthrough and macos bigsur with igpu gvt-d passthrough (using this https://github.com/patmagauran/i915ovmfPkg vbios compiled by myself, if I use the pre-compile one does not work)

Unfortunately whenever I start the 2nd vm, the host will crash and reboot itself. (Does not matter whether I start win10 first or big sur first)

If I start big sur without igpu passthrough (diplay via vnc/vmware), both vms run just fine

There is no crash report in syslog
11.19.46 - before crash
11.21.29 - host starting after reboot

My System:
Asus z390i
Intel i9 9900k
RX6900xt
Multimonitor in bios enabled
Host Boot using igpu
Both passthrough have dedicated monitor
Linux home 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 GNU/Linux

Code: 
Feb  9 11:19:45 home pvedaemon[1136]: <root@pam> starting task UPID:home:000006E4:00004C34:60220D61:qmstart:101:root@pam:
Feb  9 11:19:45 home pvedaemon[1764]: start VM 101: UPID:home:000006E4:00004C34:60220D61:qmstart:101:root@pam:
Feb  9 11:19:45 home pvedaemon[1764]: no efidisk configured! Using temporary efivars disk.
Feb  9 11:19:45 home kernel: [  195.208617] nvme nvme1: failed to set APST feature (-19)
Feb  9 11:19:45 home systemd[1]: Started 101.scope.
Feb  9 11:19:45 home systemd-udevd[1777]: Using default interface naming scheme 'v240'.
Feb  9 11:19:45 home systemd-udevd[1777]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  9 11:19:45 home systemd-udevd[1777]: Could not generate persistent MAC address for tap101i0: No such file or directory
Feb  9 11:19:46 home kernel: [  195.832655] device tap101i0 entered promiscuous mode
Feb  9 11:19:46 home systemd-udevd[1767]: Using default interface naming scheme 'v240'.
Feb  9 11:19:46 home systemd-udevd[1767]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  9 11:19:46 home systemd-udevd[1767]: Could not generate persistent MAC address for fwbr101i0: No such file or directory
Feb  9 11:19:46 home systemd-udevd[1777]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  9 11:19:46 home systemd-udevd[1777]: Could not generate persistent MAC address for fwpr101p0: No such file or directory
Feb  9 11:19:46 home systemd-udevd[1776]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  9 11:19:46 home systemd-udevd[1776]: Using default interface naming scheme 'v240'.
Feb  9 11:19:46 home systemd-udevd[1776]: Could not generate persistent MAC address for fwln101i0: No such file or directory
Feb  9 11:19:46 home kernel: [  195.850368] fwbr101i0: port 1(fwln101i0) entered blocking state
Feb  9 11:19:46 home kernel: [  195.850370] fwbr101i0: port 1(fwln101i0) entered disabled state
Feb  9 11:19:46 home kernel: [  195.850411] device fwln101i0 entered promiscuous mode
Feb  9 11:19:46 home kernel: [  195.850447] fwbr101i0: port 1(fwln101i0) entered blocking state
Feb  9 11:19:46 home kernel: [  195.850448] fwbr101i0: port 1(fwln101i0) entered forwarding state
Feb  9 11:19:46 home kernel: [  195.852696] vmbr0: port 3(fwpr101p0) entered blocking state
Feb  9 11:19:46 home kernel: [  195.852697] vmbr0: port 3(fwpr101p0) entered disabled state
Feb  9 11:19:46 home kernel: [  195.852739] device fwpr101p0 entered promiscuous mode
Feb  9 11:19:46 home kernel: [  195.852756] vmbr0: port 3(fwpr101p0) entered blocking state
Feb  9 11:19:46 home kernel: [  195.852757] vmbr0: port 3(fwpr101p0) entered forwarding state
Feb  9 11:19:46 home kernel: [  195.855256] fwbr101i0: port 2(tap101i0) entered blocking state
Feb  9 11:19:46 home kernel: [  195.855257] fwbr101i0: port 2(tap101i0) entered disabled state
Feb  9 11:19:46 home kernel: [  195.855311] fwbr101i0: port 2(tap101i0) entered blocking state
Feb  9 11:19:46 home kernel: [  195.855312] fwbr101i0: port 2(tap101i0) entered forwarding state
Feb  9 11:21:29 home kernel: [    0.000000] Linux version 5.4.78-2-pve (build@pve) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) ()
Feb  9 11:21:29 home kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.78-2-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on video=efifb:off i915.enable_gvt=1 i915.enable_guc=0 iommu=pt
Feb  9 11:21:29 home kernel: [    0.000000] KERNEL supported cpus:
Feb  9 11:21:29 home kernel: [    0.000000]   Intel GenuineIntel
Feb  9 11:21:29 home kernel: [    0.000000]   AMD AuthenticAMD
Feb  9 11:21:29 home kernel: [    0.000000]   Hygon HygonGenuine
Feb  9 11:21:29 home kernel: [    0.000000]   Centaur CentaurHauls

LSPCI
Code: 
00:00.0 Host bridge: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
    Subsystem: ASUSTeK Computer Inc. 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S]
    Kernel driver in use: skl_uncore
    Kernel modules: ie31200_edac
00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 0d)
    Kernel driver in use: pcieport
00:02.0 VGA compatible controller: Intel Corporation Device 3e98 (rev 02)
    Subsystem: ASUSTeK Computer Inc. Device 8694
    Kernel driver in use: vfio-pci
    Kernel modules: i915
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH USB 3.1 xHCI Host Controller
    Kernel driver in use: xhci_hcd
    Kernel modules: xhci_pci
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH Shared SRAM
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH HECI Controller
    Kernel driver in use: mei_me
    Kernel modules: mei_me
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH SATA AHCI Controller
    Kernel driver in use: ahci
    Kernel modules: ahci
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port (rev f0)
    Kernel driver in use: pcieport
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port (rev f0)
    Kernel driver in use: pcieport
00:1c.7 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port (rev f0)
    Kernel driver in use: pcieport
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port (rev f0)
    Kernel driver in use: pcieport
00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10)
    Subsystem: ASUSTeK Computer Inc. Z390 Chipset LPC/eSPI Controller
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH cAVS
    Kernel modules: snd_hda_intel, snd_sof_pci
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH SMBus Controller
    Kernel driver in use: i801_smbus
    Kernel modules: i2c_i801
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
    Subsystem: ASUSTeK Computer Inc. Cannon Lake PCH SPI Controller
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
    Subsystem: ASUSTeK Computer Inc. Ethernet Connection (7) I219-V
    Kernel driver in use: e1000e
    Kernel modules: e1000e
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1478 (rev c0)
    Kernel driver in use: pcieport
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1479
    Kernel driver in use: pcieport
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 73bf (rev c0)
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0e3a
    Kernel driver in use: vfio-pci
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab28
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device ab28
    Kernel driver in use: vfio-pci
    Kernel modules: snd_hda_intel
03:00.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 73a6
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 73a6
    Kernel driver in use: vfio-pci
    Kernel modules: xhci_pci
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Device 73a4
    Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0408
    Kernel driver in use: vfio-pci
04:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. Device 5762 (rev 01)
    Subsystem: Realtek Semiconductor Co., Ltd. Device 5762
    Kernel driver in use: nvme
06:00.0 Network controller: Broadcom Limited BCM4352 802.11ac Wireless Network Adapter (rev 03)
    Subsystem: Lenovo BCM4352 802.11ac Wireless Network Adapter
    Kernel driver in use: wl
    Kernel modules: bcma, wl
07:00.0 Non-Volatile memory controller: Device 1cc1:8201 (rev 03)
    Subsystem: Device 1cc1:8201
    Kernel driver in use: nvme

Windows 10 conf
Code: 
balloon: 0
bios: ovmf
boot: order=sata0;net0
cores: 8
cpu: host
hostpci0: 03:00,pcie=1,x-vga=1
hostpci1: 07:00
machine: q35
memory: 16384
name: Win10
net0: e1000=FA:44:C9:91:AC:11,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
sata0: local-lvm:vm-101-disk-0,size=100G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=e32fa60c-84b1-4d83-ba40-9910f1ca2769
sockets: 1
vga: none
vmgenid: 14df467b-c8c7-4ae7-b21f-9f5ef6369ae8

Bigsur conf
Code: 
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -device vfio-pci,host=00:02.0,romfile=i915ovmf.rom,x-igd-opregion=on,x-vga=on
balloon: 0
bios: ovmf
boot: order=sata0
cores: 8
efidisk0: local-lvm:vm-100-disk-1,size=4M
machine: q35
memory: 16384
name: BigSur
net0: vmxnet3=1A:85:5E:66:BD:28,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: other
sata0: local-lvm:vm-100-disk-0,cache=unsafe,discard=on,size=128G,ssd=1
scsihw: virtio-scsi-pci
smbios1: uuid=68633701-66af-43e4-a01d-bea8eb75b991
sockets: 1
usb0: host=1532:0088
usb1: host=046d:c545
vga: none
vmgenid: 02a7b9fc-c839-408c-bc22-ff5acfc9bdd5
 
Last edited:
how much memory do you have in the machine? both vms have 16GiB assigned, and for pci passthrough that must be fully allocated, so you'd need at least 34GiB ram if nothing else runs on that machine
 
Yesss that's it!! the problem was memory overcommit lol.
I got 32GB ram, once big sur memory reduced to 12GB the problem's gone

thank you
 
@x01 I can't get my 8569u integrated iris655 work with Win10 clients,
both the pre-compiled one and my fresh new build.
My Win10 client machine boot up with no error and I can log in with RDP.
the iris 655 hardware driver is shown with no exclamation mark, But the HDMI has no signal.
I didn't try macOS, at my site, Linux clients work perfectly including HDMI audio.

Two questions, thanks a lot if you can share something.
1. Can you share the binary you compiled, thanks a lot?
2. Does intel gpu works in Win10 client.
 
Last edited:
sqmm said:
@x01 I can't get my 8569u integrated iris655 work with Win10 clients,
both the pre-compiled one and my fresh new build.
My Win10 client machine boot up with no error and I can log in with RDP.
the iris 655 hardware driver is shown with no exclamation mark, But the HDMI has no signal.
I didn't try macOS, at my site, Linux clients work perfectly including HDMI audio.

Two questions, thanks a lot if you can share something.
1. Can you share the binary you compiled, thanks a lot?
2. Does intel gpu works in Win10 client.
Click to expand...
1. I may upload next week since I'm out of town right now
2. Yes it works. This is the trick for win10
  • Set igpu as primary boot with x-vga=on
  • set vga to vmware
  • start the vm and control/remote everything via proxmox console
  • go to display setting you'll see windows detected 2 monitor, 1st igpu and 2nd vmware but your monitor still no sgnal/output
  • go to device manager, uninstall intel graphics, then scan hardware changes. Voila!! Your monitor will show display and audio works
Question for you, do you get POST display in your current setup? I still cannot figure out why no POST display in my case
 
Last edited:
@x01 Hi, how did you manage to passthrough igpu to mac os vm. Did it require any tricks in other config files? On vm startup I'm getting:
kvm: -device vfio-pci,host=00:02.0,romfile=i915ovmf.rom,x-igd-opregion=on,x-vga=on: vfio 0000:00:02.0: failed getting region info for VGA region index 8: Invalid argument device does not support requested feature x-vga
Click to expand...

Here is my big sur vm config:
agent: 1
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc -device vfio-pci,host=00:02.0,romfile=i915ovmf.rom,x-igd-opregion=on,x-vga=on
balloon: 0
bios: ovmf
boot: order=virtio0;net0
cores: 8
efidisk0: local-lvm:vm-105-disk-1,size=4M
machine: q35
memory: 8000
name: dev
net0: vmxnet3=BE:25:6C:38:CD:51,bridge=vmbr1
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=240be1b9-8e69-41df-afde-ca2d30bff891
sockets: 1
usb0: host=046d:c534,usb3=1
vga: vmware
virtio0: local-lvm:vm-105-disk-0,cache=unsafe,discard=on,size=100G
vmgenid: 59235866-f31d-469c-8d09-c5ac201d7e8c
 
nerderlorder said:
@x01 Hi, how did you manage to passthrough igpu to mac os vm. Did it require any tricks in other config files? On vm startup I'm getting:


Here is my big sur vm config:
agent: 1
args: -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" -smbios type=2 -device usb-kbd,bus=ehci.0,port=2 -cpu host,kvm=on,vendor=GenuineIntel,+kvm_pv_unhalt,+kvm_pv_eoi,+hypervisor,+invtsc -device vfio-pci,host=00:02.0,romfile=i915ovmf.rom,x-igd-opregion=on,x-vga=on
balloon: 0
bios: ovmf
boot: order=virtio0;net0
cores: 8
efidisk0: local-lvm:vm-105-disk-1,size=4M
machine: q35
memory: 8000
name: dev
net0: vmxnet3=BE:25:6C:38:CD:51,bridge=vmbr1
numa: 1
ostype: other
scsihw: virtio-scsi-pci
smbios1: uuid=240be1b9-8e69-41df-afde-ca2d30bff891
sockets: 1
usb0: host=046d:c534,usb3=1
vga: vmware
virtio0: local-lvm:vm-105-disk-0,cache=unsafe,discard=on,size=100G
vmgenid: 59235866-f31d-469c-8d09-c5ac201d7e8c
Click to expand...

First thing first, which rom do you use? Did you put the correct rom file to the correct path?
The pre-compiled version from the link above not work for me, I had to compile myself
Next, the rom will not display anything (no POST screen) until macos driver kicks in. Some people reported that they got POST screen on certain monitors though
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back