[SOLVED] PCI-passthrough freezes host on UEFI, but legacy BIOS works (Solved: too much RAM assigned)

[Sandbo]

I am trying to setup a PC with the following spec:

• Intel C422 chipset
• W-2225 for CPU

Running Proxmox 7.3-4, and I am installing 2 VMs with legacy BIOS for virtual routers, and 1 VM with OVMF and machine type q35 for a Windows 11 installation.
All installed well, I then followed the wiki page to setup IOMMU and the returns of the test scripts looked fine.

However, I realized if I pass any PCI-E devices to the Windows VM (with UEFI bios), it freezes the host the moment the VM was started, no display will be seen from the console and I have to hard reset the system to get it back online.
It doesn't matter which device I am trying to pass, GPU/NVMe SSD all will do the same.

As a comparison, I tried to pass the same SSD which froze the system, instead to the VM running legacy BIOS, it works without any problem and I confirmed by checking inside the guest Linux that I can see the SSD nvme partitions.

I tried to look at the journal for any hints, but it wasn't very indicative.
Below are from where I tried to start the UEFI VM with SSD passed to it.

Jan 17 16:41:19 cloverleaf pvedaemon[1188]: <root@pam> starting task UPID:cloverleaf:000017C8:00021127:63C6511F:qmstart:101:root@pam:
Jan 17 16:41:19 cloverleaf pvedaemon[6088]: start VM 101: UPID:cloverleaf:000017C8:00021127:63C6511F:qmstart:101:root@pam:
Jan 17 16:41:21 cloverleaf pveproxy[1197]: worker exit
Jan 17 16:41:21 cloverleaf systemd[1]: Started 101.scope.
Jan 17 16:41:21 cloverleaf systemd-udevd[6098]: Using default interface naming scheme 'v247'.
Jan 17 16:41:21 cloverleaf systemd-udevd[6098]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:21 cloverleaf pveproxy[1194]: worker 1197 finished
Jan 17 16:41:21 cloverleaf pveproxy[1194]: starting 1 worker(s)
Jan 17 16:41:21 cloverleaf pveproxy[1194]: worker 6125 started
Jan 17 16:41:22 cloverleaf kernel: device tap101i0 entered promiscuous mode
Jan 17 16:41:22 cloverleaf systemd-udevd[6098]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf systemd-udevd[6098]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf systemd-udevd[6092]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf systemd-udevd[6092]: Using default interface naming scheme 'v247'.
Jan 17 16:41:22 cloverleaf kernel: vmbr2: port 2(fwpr101p0) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: vmbr2: port 2(fwpr101p0) entered disabled state
Jan 17 16:41:22 cloverleaf kernel: device fwpr101p0 entered promiscuous mode
Jan 17 16:41:22 cloverleaf kernel: vmbr2: port 2(fwpr101p0) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: vmbr2: port 2(fwpr101p0) entered forwarding state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 1(fwln101i0) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 1(fwln101i0) entered disabled state
Jan 17 16:41:22 cloverleaf kernel: device fwln101i0 entered promiscuous mode
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 1(fwln101i0) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 1(fwln101i0) entered forwarding state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 2(tap101i0) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 2(tap101i0) entered disabled state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 2(tap101i0) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i0: port 2(tap101i0) entered forwarding state
Jan 17 16:41:22 cloverleaf systemd-udevd[6148]: Using default interface naming scheme 'v247'.
Jan 17 16:41:22 cloverleaf systemd-udevd[6148]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf kernel: device tap101i1 entered promiscuous mode
Jan 17 16:41:22 cloverleaf systemd-udevd[6148]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf systemd-udevd[6148]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf systemd-udevd[6098]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Jan 17 16:41:22 cloverleaf kernel: vmbr1: port 2(fwpr101p1) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: vmbr1: port 2(fwpr101p1) entered disabled state
Jan 17 16:41:22 cloverleaf kernel: device fwpr101p1 entered promiscuous mode
Jan 17 16:41:22 cloverleaf kernel: vmbr1: port 2(fwpr101p1) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: vmbr1: port 2(fwpr101p1) entered forwarding state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 1(fwln101i1) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 1(fwln101i1) entered disabled state
Jan 17 16:41:22 cloverleaf kernel: device fwln101i1 entered promiscuous mode
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 1(fwln101i1) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 1(fwln101i1) entered forwarding state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 2(tap101i1) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 2(tap101i1) entered disabled state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 2(tap101i1) entered blocking state
Jan 17 16:41:22 cloverleaf kernel: fwbr101i1: port 2(tap101i1) entered forwarding state
Jan 17 16:41:30 cloverleaf pvedaemon[1187]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Jan 17 16:41:31 cloverleaf pvedaemon[1186]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Jan 17 16:41:31 cloverleaf pveproxy[1195]: proxy detected vanished client connection
Jan 17 16:41:34 cloverleaf pvestatd[1158]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Jan 17 16:41:37 cloverleaf pvestatd[1158]: status update time (11.468 seconds)
Jan 17 16:41:39 cloverleaf pvedaemon[1187]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 51 retries
Jan 17 16:41:39 cloverleaf pveproxy[1195]: proxy detected vanished client connection
Jan 17 16:41:44 cloverleaf pveproxy[1196]: proxy detected vanished client connection
Jan 17 16:41:48 cloverleaf pvestatd[1158]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 39 retries
Jan 17 16:41:51 cloverleaf pvedaemon[1187]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 38 retries
Jan 17 16:41:52 cloverleaf pvedaemon[1188]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 37 retries
Jan 17 16:41:53 cloverleaf pveproxy[1195]: proxy detected vanished client connection
Jan 17 16:41:53 cloverleaf pvedaemon[6088]: start failed: command '/usr/bin/kvm -id 101 -name 'cloverleafmeas,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server=on,>
Jan 17 16:41:53 cloverleaf pvedaemon[1188]: <root@pam> end task UPID:cloverleaf:000017C8:00021127:63C6511F:qmstart:101:root@pam: start failed: command '/usr/bin/kvm -id 101 -name 'cloverleafmeas,debug-threads=>
Jan 17 16:41:56 cloverleaf pvestatd[1158]: status update time (18.297 seconds)
Jan 17 16:42:01 cloverleaf pvedaemon[1188]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 49 retries
Jan 17 16:42:03 cloverleaf pvedaemon[1188]: <root@pam> starting task UPID:cloverleaf:000018BD:0002220F:63C6514A:vncproxy:101:root@pam:
Jan 17 16:42:03 cloverleaf pvedaemon[6333]: starting vnc proxy UPID:cloverleaf:000018BD:0002220F:63C6514A:vncproxy:101:root@pam:
Jan 17 16:42:05 cloverleaf pvestatd[1158]: VM 101 qmp command failed - VM 101 qmp command 'query-proxmox-support' failed - unable to connect to VM 101 qmp socket - timeout after 40 retries
Jan 17 16:42:33 cloverleaf pvestatd[1158]: storage 'B310' is not online
Jan 17 16:42:44 cloverleaf pvestatd[1158]: got timeout
Jan 17 16:42:54 cloverleaf pvestatd[1158]: unable to activate storage 'CloverleafBKUP_B310' - directory '/mnt/pve/B310/labPC_backup/Triton-Cloverleaf' does not exist or is unreachable
Jan 17 16:42:55 cloverleaf pvestatd[1158]: status update time (59.589 seconds)
Jan 17 16:42:56 cloverleaf pve-firewall[1157]: firewall update time (39.612 seconds)
Jan 17 16:43:36 cloverleaf pve-firewall[1157]: firewall update time (39.442 seconds)
Jan 17 16:43:42 cloverleaf pve-ha-lrm[1202]: loop take too long (35 seconds)

 

[dcsapak]

can you post the iommu groups and the vm config? also the complete 'dmesg' output would be interesting

 

[Sandbo]

can you post the iommu groups and the vm config? also the complete 'dmesg' output would be interesting

Thanks for getting back to me, I will post them when I have a chance to get back to the server (might be away from it for a while).

I would like to update, though, with more tests done earlier, I realized I could forward the said SSD (and possibly other PCI devices, not tested), to a new UEFI Guest booting from a ubuntu 22.04 ISO. I was trying to compare the setting between the new VM which worked to the Windows VM which didn't, but had no clue yet.

So apparently UEFI is not the reason, but a misconfiguration within the Windows VM has blocked PCI passthrough from working. I will update again if I have any other findings.

 

[Sandbo]

Second update: The issue is now solved, turns out I assigned too much memory to the Windows VM.

I have 32 GB RAM, and I assigned 32GB Max/16GB Min to the Windows guest, this way it won't boot if I have a PCI-E device attached.
As the Ubuntu guest works, I tried to change the setting of the Windows VM to follow ubuntu, and it worked as I reduce the assignment to 16 GB only. Then assigning 24GB Max/16GB Min to Windows VM also worked and I can pass the SSD. Interesting.

 

[leesteken]

PCI(e) passthrough requires all VM memory to be pinned into actual host memory. Ballooning is therefore also not possible, so there is no need to set the minumum memory lower than the maximum memory.

 

[Sandbo]

PCI(e) passthrough requires all VM memory to be pinned into actual host memory. Ballooning is therefore also not possible, so there is no need to set the minumum memory lower than the maximum memory.

Thanks, this explains it really well; as the only system that did not work was the 32GB RAM guest I tried - other guests were assigned much less memory. So it wasn't freezing because of any conflicts but solely because of the system was running out of RAM.