PCI Passthrough issues Proxmox4

A

adamb

Famous Member
Proxmox Subscriber
Mar 1, 2012
1,323
73
113
We are running into an issue with pci pass through on Proxmox4.

Just passing some HBA's to a VM which we have done in the past with no issues.

The passthrough works, but only when the VM has less than 70G of ram. Anything above 70G and the VM will fail to start. Below is the message when it fails.

TASK ERROR: start failed: command '/usr/bin/systemd-run --scope --slice qemu --unit 100 -p 'KillMode=none' -p 'CPUShares=1000' /usr/bin/kvm -id 100 -chardev 'socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios 'type=1,uuid=5eceeeef-abaa-469c-813a-95eeab1638fe' -drive 'if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd' -drive 'if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd' -name monthvault -smp '46,sockets=2,cores=23,maxcpus=46' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000' -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 80000 -object 'memory-backend-ram,size=40000M,id=ram-node0' -numa 'node,nodeid=0,cpus=0-22,memdev=ram-node0' -object 'memory-backend-ram,size=40000M,id=ram-node1' -numa 'node,nodeid=1,cpus=23-45,memdev=ram-node1' -k en-us -device 'pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f' -device 'pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e' -device 'piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2' -device 'usb-tablet,id=tablet,bus=uhci.0,port=1' -device 'vfio-pci,host=08:00.0,id=hostpci0,bus=pci.0,addr=0x10' -device 'vfio-pci,host=81:00.0,id=hostpci1,bus=pci.0,addr=0x11' -device 'vfio-pci,host=88:00.0,id=hostpci2,bus=pci.0,addr=0x1b' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:2b747e7588ec' -device 'lsi,id=scsihw0,bus=pci.0,addr=0x5' -drive 'file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on' -device 'scsi-hd,bus=scsihw0.0,scsi-id=0,drive=drive-scsi0,id=scsi0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=36:63:36:33:35:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300'' failed: got timeout

root@monthvaultprox:/var/log# pveversion -v
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-28
pve-kernel-4.2.2-1-pve: 4.2.2-16
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-42
pve-firmware: 1.1-7
libpve-common-perl: 4.0-42
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-18
pve-container: 1.0-35
pve-firewall: 2.0-14
pve-ha-manager: 1.0-16
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie
 
Even more interesting. When starting the VM with more than 70G its starting 2 instances of the same machine.

root@monthvaultprox:~# ps -ef | grep kvm
root 1371 2 0 07:41 ? 00:00:00 [kvm-irqfd-clean]
root 2321 2263 0 07:43 ? 00:00:00 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=5eceeeef-abaa-469c-813a-95eeab1638fe -drive if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd -drive if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd -name monthvault -smp 46,sockets=2,cores=23,maxcpus=46 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 330000 -object memory-backend-ram,size=165000M,id=ram-node0 -numa node,nodeid=0,cpus=0-22,memdev=ram-node0 -object memory-backend-ram,size=165000M,id=ram-node1 -numa node,nodeid=1,cpus=23-45,memdev=ram-node1 -k en-us -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device vfio-pci,host=08:00.0,id=hostpci0,bus=pci.0,addr=0x10 -device vfio-pci,host=81:00.0,id=hostpci1,bus=pci.0,addr=0x11 -device vfio-pci,host=88:00.0,id=hostpci2,bus=pci.0,addr=0x1b -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:2b747e7588ec -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=36:63:36:33:35:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
root 2323 2321 0 07:43 ? 00:00:00 [kvm] <defunct>
root 2324 1 99 07:43 ? 00:00:07 /usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/100.vnc,x509,password -pidfile /var/run/qemu-server/100.pid -daemonize -smbios type=1,uuid=5eceeeef-abaa-469c-813a-95eeab1638fe -drive if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd -drive if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd -name monthvault -smp 46,sockets=2,cores=23,maxcpus=46 -nodefaults -boot menu=on,strict=on,reboot-timeout=1000 -vga cirrus -cpu host,+kvm_pv_unhalt,+kvm_pv_eoi -m 330000 -object memory-backend-ram,size=165000M,id=ram-node0 -numa node,nodeid=0,cpus=0-22,memdev=ram-node0 -object memory-backend-ram,size=165000M,id=ram-node1 -numa node,nodeid=1,cpus=23-45,memdev=ram-node1 -k en-us -device pci-bridge,id=pci.1,chassis_nr=1,bus=pci.0,addr=0x1e -device pci-bridge,id=pci.2,chassis_nr=2,bus=pci.0,addr=0x1f -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device vfio-pci,host=08:00.0,id=hostpci0,bus=pci.0,addr=0x10 -device vfio-pci,host=81:00.0,id=hostpci1,bus=pci.0,addr=0x11 -device vfio-pci,host=88:00.0,id=hostpci2,bus=pci.0,addr=0x1b -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -iscsi initiator-name=iqn.1993-08.org.debian:01:2b747e7588ec -device virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5 -drive file=/var/lib/vz/images/100/vm-100-disk-1.raw,if=none,id=drive-scsi0,format=raw,cache=none,aio=native,detect-zeroes=on -device scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=100 -netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=36:63:36:33:35:38,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
root 2385 2 0 07:43 ? 00:00:00 [kvm-pit/2324]
root 2387 2202 0 07:43 pts/0 00:00:00 grep kvm
 
Making some progress. I can now give the VM 107G before I start to hit the same issue. One thing to note is it seems to take alot of time reserving the VM memory when you start the VM. As you give it more ram it takes longer and longer to reserve. It almost seems like a timing issue of some sort, but I am unsure.

At 107G it takes roughly 30-45 seconds before you get a bios on the console.
 
adamb said:
Making some progress. I can now give the VM 107G before I start to hit the same issue. One thing to note is it seems to take alot of time reserving the VM memory when you start the VM. As you give it more ram it takes longer and longer to reserve. It almost seems like a timing issue of some sort, but I am unsure.

At 107G it takes roughly 30-45 seconds before you get a bios on the console.
Click to expand...

try the last kernel from pvetest repository, they are big pci passthrough bugfixes related to your problem.
 
spirit said:
for memory, windows fill memory with zeroes at start.
you can try to enable numa, it should help. (If I remember, windows don't fill memory when numa is enabled)
Click to expand...


I was using numa before upgrading the kernel. Guest OS is CentOS7.2
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back