Hi everyone,
Trying to wrap my head around an issue where a test VM when being attached with a virtual Infiniband adapter is not starting. If I remove the virtual interface, it will start.
The VM seems to start initially, but the console never connects. Shortly after, the VM shuts down. I've attached the error log from Proxmox below.
TASK ERROR: start failed: command '/usr/bin/kvm -id 120 -name 'ib-test,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/120.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/120.pid -daemonize -smbios 'type=1,uuid=41be0538-dabf-4208-96a0-c0b7654149d2' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/120.vnc,password=on' -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=c1628483-f954-4161-9595-e12441157ade' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=0000:86:00.3,id=hostpci0,bus=pci.0,addr=0x10' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:a054d3286d9' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/hpc-next-gen-zfs/vm-120-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap120i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=22:2B:B3:24:3C:98,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=1024' -machine 'type=pc+pve0'' failed: got timeout
The host is properly configured with IOMMU and SR-IOV because there is another VM with a working virtual IB interface. That VM is on Rocky Linux 8.
If I take the same virtual interface that fails to start the test VM and attach it to a different Centos 8 VM, it starts properly. That Centos VM has no installed IB drivers or packages.
For my test VM, I've tried attaching the virtual interface with both Ubuntu 22 and Rocky Linux 8 installed without success. The qm.conf file is near identical (aside from a few VM specific lines) on the test VMs and the VMs where the IB interface works.
The virtual interfaces that I am attaching are active on the host under the networking tab.
Anyone have any ideas?
Trying to wrap my head around an issue where a test VM when being attached with a virtual Infiniband adapter is not starting. If I remove the virtual interface, it will start.
The VM seems to start initially, but the console never connects. Shortly after, the VM shuts down. I've attached the error log from Proxmox below.
TASK ERROR: start failed: command '/usr/bin/kvm -id 120 -name 'ib-test,debug-threads=on' -no-shutdown -chardev 'socket,id=qmp,path=/var/run/qemu-server/120.qmp,server=on,wait=off' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/120.pid -daemonize -smbios 'type=1,uuid=41be0538-dabf-4208-96a0-c0b7654149d2' -smp '4,sockets=1,cores=4,maxcpus=4' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vnc 'unix:/var/run/qemu-server/120.vnc,password=on' -cpu kvm64,enforce,+kvm_pv_eoi,+kvm_pv_unhalt,+lahf_lm,+sep -m 4096 -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'vmgenid,guid=c1628483-f954-4161-9595-e12441157ade' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=0000:86:00.3,id=hostpci0,bus=pci.0,addr=0x10' -device 'VGA,id=vga,bus=pci.0,addr=0x2' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3,free-page-reporting=on' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:a054d3286d9' -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/dev/zvol/hpc-next-gen-zfs/vm-120-disk-0,if=none,id=drive-scsi0,format=raw,cache=none,aio=io_uring,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap120i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=22:2B:B3:24:3C:98,netdev=net0,bus=pci.0,addr=0x12,id=net0,rx_queue_size=1024,tx_queue_size=1024' -machine 'type=pc+pve0'' failed: got timeout
The host is properly configured with IOMMU and SR-IOV because there is another VM with a working virtual IB interface. That VM is on Rocky Linux 8.
If I take the same virtual interface that fails to start the test VM and attach it to a different Centos 8 VM, it starts properly. That Centos VM has no installed IB drivers or packages.
For my test VM, I've tried attaching the virtual interface with both Ubuntu 22 and Rocky Linux 8 installed without success. The qm.conf file is near identical (aside from a few VM specific lines) on the test VMs and the VMs where the IB interface works.
The virtual interfaces that I am attaching are active on the host under the networking tab.
Anyone have any ideas?
Last edited: