PCI Passthrough issues Proxmox4

A

adamb

Famous Member
Proxmox Subscriber
Mar 1, 2012
1,329
78
113
We are running into an issue with pci pass through on Proxmox4.

Just passing some HBA's to a VM which we have done in the past with no issues.

The passthrough works, but only when the VM has less than 70G of ram. Anything above 70G and the VM will fail to start. Below is the message when it fails.

TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 100 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=5eceeeef-abaa-469c-813a-95eeab1638fe' -drive 'if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd' -drive 'if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd' -name monthvault -smp '46,sockets=2,cores=23,maxcpus=46' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 80000 -object 'memory-backend-ram,size=40000M,id=ram-node0' -numa 'node,nodeid=0,cpus=0-22,memdev=ram-node0' -object 'memory-backend-ram,size=40000M,id=ram-node1' -numa 'node,nodeid=1,cpus=23-45,memdev=ram-node1' -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=08:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'vfio-pci,host=81:00.0,id=hostpci1,bus=pci.0,addr=0x11' -device 'vfio-pci,host=88:00.0,id=hostpci2,bus=pci.0,addr=0x1b' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:2b747e7588ec' -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,scsi-id=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=36:63:36:33:35:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: got timeout

root@monthvaultprox:/var/log# pveversion -v
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-28
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-42
pve-firmware: 1.1-7
libpve-common-perl: 4.0-42
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-18
pve-container: 1.0-35
pve-firewall: 2.0-14
pve-ha-manager: 1.0-16
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie
 
Even more interesting. When starting the VM with more than 70G its starting 2 instances of the same machine.

root@monthvaultprox:~# ps -ef | grep kvm
root 1371 2 0 07:41 ? 00:00:00 [kvm-irqfd-clean]
root 2321 2263 0 07:43 ? 00:00:00 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=5eceeeef-abaa-469c-813a-95eeab1638fe -drive if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd -drive if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd -name monthvault -smp 46,sockets=2,cores=23,maxcpus=46 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 330000 -object memory-backend-ram,size=165000M,id=ram-node0 -numa node,nodeid=0,cpus=0-22,memdev=ram-node0 -object memory-backend-ram,size=165000M,id=ram-node1 -numa node,nodeid=1,cpus=23-45,memdev=ram-node1 -k en-us -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device vfio-pci,host=08:00.0,id=hostpci0,bus=pci.0,addr=0x10 -device vfio-pci,host=81:00.0,id=hostpci1,bus=pci.0,addr=0x11 -device vfio-pci,host=88:00.0,id=hostpci2,bus=pci.0,addr=0x1b -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:2b747e7588ec -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=36:63:36:33:35:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
root 2323 2321 0 07:43 ? 00:00:00 [kvm] <defunct>
root 2324 1 99 07:43 ? 00:00:07 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=5eceeeef-abaa-469c-813a-95eeab1638fe -drive if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd -drive if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd -name monthvault -smp 46,sockets=2,cores=23,maxcpus=46 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 330000 -object memory-backend-ram,size=165000M,id=ram-node0 -numa node,nodeid=0,cpus=0-22,memdev=ram-node0 -object memory-backend-ram,size=165000M,id=ram-node1 -numa node,nodeid=1,cpus=23-45,memdev=ram-node1 -k en-us -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device vfio-pci,host=08:00.0,id=hostpci0,bus=pci.0,addr=0x10 -device vfio-pci,host=81:00.0,id=hostpci1,bus=pci.0,addr=0x11 -device vfio-pci,host=88:00.0,id=hostpci2,bus=pci.0,addr=0x1b -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:2b747e7588ec -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=36:63:36:33:35:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
root 2385 2 0 07:43 ? 00:00:00 [kvm-pit/2324]
root 2387 2202 0 07:43 pts/0 00:00:00 grep kvm
 
Making some progress. I can now give the VM 107G before I start to hit the same issue. One thing to note is it seems to take alot of time reserving the VM memory when you start the VM. As you give it more ram it takes longer and longer to reserve. It almost seems like a timing issue of some sort, but I am unsure.

At 107G it takes roughly 30-45 seconds before you get a bios on the console.
 
adamb said:
Making some progress. I can now give the VM 107G before I start to hit the same issue. One thing to note is it seems to take alot of time reserving the VM memory when you start the VM. As you give it more ram it takes longer and longer to reserve. It almost seems like a timing issue of some sort, but I am unsure.

At 107G it takes roughly 30-45 seconds before you get a bios on the console.
Click to expand...

try the last kernel from pvetest repository, they are big pci passthrough bugfixes related to your problem.
 
You must log in or register to reply here.
Top Bottom