Vm's won't start most of the time

[Jsingh]

I am starting with a new proxmox installation with the following configuration:

Threadripper 1950X
Gigabyte X399 Auros Extreme
Geforce Gtx 750
Geforce Gtx 9800 Gtx+
2 X 500 Gb Adata SX8200 Pro
3 X 1Tb HDD for daily backup

I have created two VM's each having one of the cards passed through it. I am facing two issues:

1. The 9800 GTX gives code 43 error even after passing the ROM File.
2. The VMs don't start 70% of the time mostly when running together

The error I get is : TASK ERROR: start failed:...... failed: got a timeout

or sometimes I proxmox will show that there is an internal error in the VM

 

[Nemesiz]

In console #qm showcmd ID
Then copy results and try to start VM from console for debugging

 

[Jsingh]

I get this and this only happens when I pass through my second GPU:

start failed: command '/usr/bin/kvm -id 101 -name DesignVm2 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' -mon 'chardev=qmp-event,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=0c286fb4-0940-4476-b331-a4173f6798dc' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=qcow2,id=drive-efidisk0,file=/mnt/VmStore/images/101/vm-101-disk-1.qcow2' -smp '6,sockets=1,cores=6,maxcpus=6' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+pcid,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' -m 8192 -object 'memory-backend-ram,id=ram-node0,size=8192M' -numa 'node,nodeid=0,cpus=0-5,memdev=ram-node0' -object 'iothread,id=iothread-virtioscsi0' -device 'vmgenid,guid=a7cc9349-1a65-4f2e-a88d-4d89082092b9' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=0a:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:19663b5e5e4b' -device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' -drive 'file=/mnt/VmStore/images/101/vm-101-disk-0.qcow2,if=none,id=drive-scsi0,cache=writeback,discard=on,format=qcow2,aio=threads,detect-zeroes=unmap' -device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=6E:39:1C:C8:84:A1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard' -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'' failed: got timeout​

 

[Nemesiz]

Run this comment directly to console (ssh).
This is the same command I just removed daemonize option

/usr/bin/kvm \
-id 101 \
-name DesignVm2 \
-chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' \
-mon 'chardev=qmp,mode=control' \
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
-mon 'chardev=qmp-event,mode=control' \
-pidfile /var/run/qemu-server/101.pid \
-smbios 'type=1,uuid=0c286fb4-0940-4476-b331-a4173f6798dc' \
-drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-drive 'if=pflash,unit=1,format=qcow2,id=drive-efidisk0,file=/mnt/VmStore/images/101/vm-101-disk-1.qcow2' \
-smp '6,sockets=1,cores=6,maxcpus=6' \
-nodefaults \
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
-vga none \
-nographic \
-no-hpet \
-cpu 'host,+pcid,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' \
-m 8192 \
-object 'memory-backend-ram,id=ram-node0,size=8192M' \
-numa 'node,nodeid=0,cpus=0-5,memdev=ram-node0' \
-object 'iothread,id=iothread-virtioscsi0' \
-device 'vmgenid,guid=a7cc9349-1a65-4f2e-a88d-4d89082092b9' \
-readconfig /usr/share/qemu-server/pve-q35.cfg \
-device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
-device 'vfio-pci,host=0a:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' \
-chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' \
-device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
-device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
-device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:19663b5e5e4b' \
-device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' \
-drive 'file=/mnt/VmStore/images/101/vm-101-disk-0.qcow2,if=none,id=drive-scsi0,cache=writeback,discard=on,format=qcow2,aio=threads,detect-zeroes=unmap' \
-device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100' \
-netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
-device 'virtio-net-pci,mac=6E:39:1C:C8:84:A1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' \
-rtc 'driftfix=slew,base=localtime' \
-machine 'type=q35' \
-global 'kvm-pit.lost_tick_policy=discard' \
-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off

 

[Jsingh]

Yeah This works! How do I make this permanent in the system?

 

[Nemesiz]

Don`t know why Proxmox gets timeout but you can add daemonize and KVM will run in background.

/usr/bin/kvm \
-id 101 \
-name DesignVm2 \
-chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' \
-mon 'chardev=qmp,mode=control' \
-chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
-mon 'chardev=qmp-event,mode=control' \
-pidfile /var/run/qemu-server/101.pid \
-daemonize \
-smbios 'type=1,uuid=0c286fb4-0940-4476-b331-a4173f6798dc' \
-drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' \
-drive 'if=pflash,unit=1,format=qcow2,id=drive-efidisk0,file=/mnt/VmStore/images/101/vm-101-disk-1.qcow2' \
-smp '6,sockets=1,cores=6,maxcpus=6' \
-nodefaults \
-boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
-vga none \
-nographic \
-no-hpet \
-cpu 'host,+pcid,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,kvm=off' \
-m 8192 \
-object 'memory-backend-ram,id=ram-node0,size=8192M' \
-numa 'node,nodeid=0,cpus=0-5,memdev=ram-node0' \
-object 'iothread,id=iothread-virtioscsi0' \
-device 'vmgenid,guid=a7cc9349-1a65-4f2e-a88d-4d89082092b9' \
-readconfig /usr/share/qemu-server/pve-q35.cfg \
-device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
-device 'vfio-pci,host=0a:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0' \
-chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' \
-device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
-device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
-device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
-iscsi 'initiator-name=iqn.1993-08.org.debian:01:19663b5e5e4b' \
-device 'virtio-scsi-pci,id=virtioscsi0,bus=pci.3,addr=0x1,iothread=iothread-virtioscsi0' \
-drive 'file=/mnt/VmStore/images/101/vm-101-disk-0.qcow2,if=none,id=drive-scsi0,cache=writeback,discard=on,format=qcow2,aio=threads,detect-zeroes=unmap' \
-device 'scsi-hd,bus=virtioscsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,rotation_rate=1,bootindex=100' \
-netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
-device 'virtio-net-pci,mac=6E:39:1C:C8:84:A1,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' \
-rtc 'driftfix=slew,base=localtime' \
-machine 'type=q35' \
-global 'kvm-pit.lost_tick_policy=discard' \
-cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off

 

[oguz]

Don`t know why Proxmox gets timeout but you can add daemonize and KVM will run in background.

we just have timeout logic for a lot of our commands (start, stop etc.) if qemu doesn't respond or something like that (also present in lxc pct commands for example).

i might look into a fix for this.. can you suggest anything maybe?

 

[Nemesiz]

make time option controllable from console/GUI for timeout commands.

 

[oguz]

we already have that in most of them with --timeout option on cli

 

[Nemesiz]

Ok but it may be settable permanent to all runs.

 

[Jsingh]

So I replaced the 9800 GTX with a geforce 1050Ti so now both Cards are UEFI compatible. I passed through both cards to different VMs and Windows installed the Basic Display adapter sucessfully.

Then I installed the driver and the moment I installed the VGA driver, the VM with Geforce 750(set as boot Graphics device in the bios installed in PCIe slot 1) froze. When I tried to stop the VM from the VM Manager the host crashed.

I restarted the host now the second VM won't start when the first is started giving a timeout error.

Don`t know why Proxmox gets timeout but you can add daemonize and KVM will run in background.

How can I do that? The Virtual Machines will be used by specific users who are not technical so I need to be able to start the VM from GUI.

 

[Jsingh]

[Update] So I had enabled memory balloon in both of Vm's. So My config was:

VM1: min: 8Gb max: 8gb
VM2: min: 4Gb max: 8Gb

I changed the min on Vm2 to 8 Gb and the Vm's are behaving correctly now.
My theory is:
1. The non uefi 9800 GTX was causing the timeout issues initially.
2. The memory balloon was causing instability and timeout errors.

Is this a known issue? Will it create issues when I pass through the SATA controller and a nvme controller to another VM I am planning to deploy?

I am going to watch the stability of the system over the next few days and see how it behaves.