[SOLVED] pcipasshrough not working after upgrade to proxmox 9

[proxman4]

Hi folks. Hope you're doing well.

I got a weird issue.

I got 2 proxmox prod-pi and pra-pi .

Each one got a LSI SAS3008 Broadcaom used as pcipassthrough by a truenas vm.

04:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 [1000:0097] (rev 02)
        Subsystem: Broadcom / LSI SAS9300-8e [1000:30a0]
        Kernel driver in use: vfio-pci
        Kernel modules: mpt3sas

Both pcie cards are connected to the same DiskBay

The goal is to have a second server in case something goes wrong with the first one and to also test upgrades.

This has been setup on proxmox 8 and it worked like a charm for both prod-pi and pra-pi

Recently i upgraded to proxmox 9.

On pra-pi the vm still works

BUT on prod-pi that is not possible to start the vm.

I triple checked everything several time

I also tried different conf on prod-pi to see if it can help.

• switch to q35
• changing cpu to host

here is the vm conf on pra-pi ( the working proxmox and truenas vm)

agent: 1
balloon: 0
bios: ovmf
boot: order=virtio0
cores: 6
cpu: x86-64-v2-AES,flags=-hv-evmcs;+aes
efidisk0: local-zfs:vm-244-disk-0,efitype=4m,size=1M
hostpci0: 0000:04:00,rombar=0
memory: 42000
meta: creation-qemu=8.1.5,ctime=1730815873
name: truenas-lab
net0: virtio=BC:24:11:5C:F6:60,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsihw: virtio-scsi-single
serial0: socket
smbios1: uuid=26383b29-8da5-4743-b5c4-cb1783e610c5
sockets: 2
tags: lab
vga: qxl
virtio0: local-zfs:vm-244-disk-1,discard=on,size=50G
vmgenid: 0996cbd3-fdda-4d02-a7d0-1c16558d2f37

Here is the log on the non-working proxmox prod-pi when starting vm hang...

start failed: command '/usr/bin/kvm -id 244 -name 'truenas-lab,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/244.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect-ms=5000' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/244.pid -daemonize -smbios 'type=1,uuid=26383b29-8da5-4743-b5c4-cb1783e610c5' -object '{"id":"throttle-drive-efidisk0","limits":{},"qom-type":"throttle-group"}' -blockdev '{"driver":"raw","file":{"driver":"file","filename":"/usr/share/pve-edk2-firmware//OVMF_CODE_4M.secboot.fd"},"node-name":"pflash0","read-only":true}' -blockdev '{"detect-zeroes":"on","discard":"ignore","driver":"throttle","file":{"cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"raw","file":{"aio":"io_uring","cache":{"direct":false,"no-flush":false},"detect-zeroes":"on","discard":"ignore","driver":"host_device","filename":"/dev/zvol/rpool/data/vm-244-disk-0","node-name":"e511f127d24a81ca85ac315de32e665","read-only":false},"node-name":"f511f127d24a81ca85ac315de32e665","read-only":false,"size":540672},"node-name":"drive-efidisk0","read-only":false,"throttle-group":"throttle-drive-efidisk0"}' -smp '12,sockets=2,cores=6,maxcpus=12' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/244.vnc,password=on' -cpu host,+kvm_pv_eoi,+kvm_pv_unhalt -m 42000 -object '{"id":"throttle-drive-virtio0","limits":{},"qom-type":"throttle-group"}' -global 'ICH9-LPC.disable_s3=1' -global 'ICH9-LPC.disable_s4=1' -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg -device 'vmgenid,guid=108cb753-144f-4b76-86de-ef9fc79ef1a1' -device 'vfio-pci,host=0000:81:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0' -chardev 'socket,id=serial0,path=/var/run/qemu-server/244.serial0,server=on,wait=off' -device 'isa-serial,chardev=serial0' -device 'qxl-vga,id=vga,max_outputs=4,bus=pcie.0,addr=0x1' -device 'virtio-serial,id=spice,bus=pci.0,addr=0x9' -chardev 'spicevmc,id=vdagent,name=vdagent' -device 'virtserialport,chardev=vdagent,name=com.redhat.spice.0' -spice 'tls-port=61008,addr=127.0.0.1,tls-ciphers=HIGH,seamless-migration=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:12726b489e3' -blockdev '{"detect-zeroes":"unmap","discard":"unmap","driver":"throttle","file":{"cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"raw","file":{"aio":"io_uring","cache":{"direct":true,"no-flush":false},"detect-zeroes":"unmap","discard":"unmap","driver":"host_device","filename":"/dev/zvol/rpool/data/vm-244-disk-1","node-name":"e92ad33ab45f02d05853199b6580e9d","read-only":false},"node-name":"f92ad33ab45f02d05853199b6580e9d","read-only":false},"node-name":"drive-virtio0","read-only":false,"throttle-group":"throttle-drive-virtio0"}' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100,write-cache=on' -netdev 'type=tap,id=net0,ifname=tap244i0,script=/usr/libexec/qemu-server/pve-bridge,downscript=/usr/libexec/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=BC:24:11:5C:F6:60,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=256,host_mtu=1500' -machine 'pflash0=pflash0,pflash1=drive-efidisk0,hpet=off,type=q35+pve0' -d guest_errors,unimp' failed: got timeout

I'm a little puzzled why it does not work on prod-pi as i didn't change anything hardware or proxmox conf.

If someone can help my poor soul by giving me a hint it would be terrific.

Thank you very much and have a good day

 

[leesteken]

A timeout without an actual error message usually points to a lack of available memory. A VM with passthrough need to pin all VM memory into actual host RAM. PVE 9 also upped the ZFS default ARC maximum (which PVE 8 reduced a lot from the default). Try reducing the ZFS ARC ( https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage ) or starting the VM with much less memory (as a test).

 

[proxman4]

Thanks @leesteken . I reduced the RAM and............. It works ! I'll definitely reduce ARC usage as server has 256GiB RAM and can handle all vm smoothly.

Thank you very much have an excellent day !