Code 43 and no image with GTX 660 and latest proxmox/pve-kernel

F

FuriousGeorge

Renowned Member
Sep 25, 2012
84
2
73
I've followed the famous "vfio tips and tricks" blog up until the 4th installment, which is distro specific.

My VM is able to recognize my GPU, I believe the problem is that my GPU /IS/ able to recognize my VM, as per: http://vfio.blogspot.com/2014/08/vfiovga-faq.html (see question 10).

I cannot manually set a vmid.conf that invokes kvm as is suggested by the FAQ (e.g. I cannot use 440 chipset, and the hyper_v extensions are always included, but should not be) but based on "qm showcmd 100" I'm trying to manually generate a working command, and work backwards from there. What I have now is:

/usr/bin/kvm -id 100 -chardev socket,id=qmp,path=/var/run/qemu-server/100.qmp,server,nowait \
-mon chardev=qmp,mode=control \
-pidfile /var/run/qemu-server/100.pid \
-daemonize \
-smbios type=1,uuid=96ca332e-4376-49de-b272-5dacc6f01335 \
-drive if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/100-OVMF_VARS-pure-efi.fd \
-name test \
-smp 8,sockets=1,cores=8,maxcpus=8 \
-nodefaults \
-boot menu=on,strict=on,reboot-timeout=1000 \
-vnc unix:/var/run/qemu-server/100.vnc,x509,password \
-no-hpet \
-cpu host,kvm=off,+kvm_pv_eoi,hv_vendor_id=Nvidia43FIX \
-m 8196 \
-k en-us \
-readconfig /usr/share/qemu-server/pve-q35.cfg \
-device usb-tablet,id=tablet,bus=ehci.0,port=1 \
-device vfio-pci,host=04:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 \
-iscsi initiator-name=iqn.1993-08.org.debian:01:3f1e9afe6fdb \
-drive file=/dev/zvol/rpool/data/vm-100-disk-1,if=none,id=drive-virtio0,cache=unsafe,format=raw,aio=threads,detect-zeroes=on \
-device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200 \
-netdev type=tap,id=net0,ifname=tap100i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on -device virtio-net-pci,mac=32:63:66:65:35:36,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 \
-rtc driftfix=slew,base=localtime \
-machine type=q35 \
-global kvm-pit.lost_tick_policy=discard \
-vga std​

Unless I misread the FAQ, the -cpu argument should be enough to hide KVM from nvidia.

Before the nvidia driver has a chance to detect it is in a vm however, I still don't see the BIOS splash screen on the GPU's display. This makes me wonder if there is not a secondary problem.

I've read that I should turn -vga to none, however, due to a bug in Proxmox, the OVMF bios does not persist between boots, so I must manually set the boot device each time, and I need a screen to do that.

There is a mailing list for vfio, and the guy who maintains the "tips and tricks" blog made some suggestions:

1) Use OVMF.
So I created a new VM as such.

2) Use pc, not q35
This appears to not be supported in proxmox, despite what the vm.conf manpage says. I think I'm missing a config file for the platform.

3) Avoid using emulated GPU.
Unfortunately, if I turn video off I cannot get into windows, as OVMF bios forgets boot order and goes to shell. Still, as I mentioned above, I feel like I should see the OVMF splash screen anyway.

Any help is much appreciated.

Here is my vm.conf:

Code: 
#hostpci1%3A  04%3A00.1,pcie=1,x-vga=on
#hostpci2%3A  05%3A00.0,pcie=1,x-vga=on
#hostpci3%3A  05%3A00.1,pcie=1,x-vga=on
bios: ovmf
boot: dcn
bootdisk: virtio0
cores: 8
cpu: host
hostpci0: 04:00.0,pcie=1
machine: q35
#machine: pc
memory: 8196
name: test
net0: virtio=32:63:66:65:35:36,bridge=vmbr0
numa: 0
ostype: win8
smbios1: uuid=96ca332e-4376-49de-b272-5dacc6f01335
sockets: 1
vga: std
virtio0: local-zfs:vm-100-disk-1,cache=unsafe,size=120G

Thanks in advance
 
Thanks for the response. I was just getting ready to update the thread.

In the course of the last few hours, I obtained a UEFI bios for my GPUs, and I was able to boot the VM with video and audio on two GPUs from the CLI, after removing all the pv_ stuff. Otherwise the VM would crash with a sad face :( if I did not remove the hyper-v stuff.

I did upgrade to the latest Proxmox at some point before getting the BIOS, but after creating the VM.

Then I cloned the VM for an unrelated reason, and like magic everything just worked from the GUI too. I've since deleted the original VM, so unfortunately I can't verify what the problem was.

For posterity, here is the my vm.conf now:

Code: 
bios: ovmf
boot: dcn
bootdisk: scsi0
cores: 8
cpu: host
hostpci0: 04:00.0,pcie=1
hostpci1: 04:00.1,pcie=1
hostpci2: 05:00.0,pcie=1
hostpci3: 05:00.1,pcie=1
machine: q35
memory: 8192
name: Office-PC-1
net0: virtio=3A:30:66:AA:61:66,bridge=vmbr0
numa: 0
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=b97ca7cc-98bf-4a68-ac83-88a26f97ee73
sockets: 1
virtio0: local-zfs:vm-110-disk-1,cache=unsafe,size=120G

Here is the invocation of kvm it generates:

Code: 
/usr/bin/kvm \
-id 110 \
-chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait \
-mon chardev=qmp,mode=control \
-pidfile /var/run/qemu-server/110.pid -daemonize -smbios type=1,uuid=b97ca7cc-98bf-4a68-ac83-88a26f97ee73 \
-drive if=pflash,format=raw,readonly,file=/usr/share/kvm/OVMF-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/110-OVMF_VARS-pure-efi.fd \
-name Office-PC-1 \
-smp 8,sockets=1,cores=8,maxcpus=8 \
-nodefaults \
-boot menu=on,strict=on,reboot-timeout=1000 \
-vga std \
-vnc unix:/var/run/qemu-server/110.vnc,x509,password \
-no-hpet \
-cpu host,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_relaxed,+kvm_pv_unhalt,+kvm_pv_eoi \
-m 8196 \
-k en-us \
-readconfig /usr/share/qemu-server/pve-q35.cfg -device usb-tablet,id=tablet,bus=ehci.0,port=1 \
-device vfio-pci,host=04:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0 \
-device vfio-pci,host=04:00.1,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0 \
-device vfio-pci,host=05:00.0,id=hostpci2,bus=ich9-pcie-port-3,addr=0x0 \
-device vfio-pci,host=05:00.1,id=hostpci3,bus=ich9-pcie-port-4,addr=0x0 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 \
-iscsi initiator-name=iqn.1993-08.org.debian:01:3f1e9afe6fdb \
-drive file=/dev/zvol/rpool/data/vm-110-disk-1,if=none,id=drive-virtio0,cache=unsafe,format=raw,aio=threads,detect-zeroes=on \
-device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa \
-netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on \
-device virtio-net-pci,mac=3A:30:66:30:61:66,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300 \
-rtc driftfix=slew,base=localtime \
-machine type=q35 \
-global kvm-pit.lost_tick_policy=discard

... it does still generate -vga std and the vnc bit ...

I did this all from the GUI, aside from adding the scsihw: and hostpciX: lines as per wiki. I don't see a big difference from what wasn't working with the previous VM. (I suppose it is proxmox inserting the commented lines at the top of the original?)

Code: 
# pveversion
pve-manager/4.2-15/6669ad2c (running kernel: 4.4.10-1-pve)


The problem with UEFI bios persistence appears to be this bug: https://bugzilla.proxmox.com/show_bug.cgi?id=977

It isn't yet apparent to me where I could get the OVMF_CODE-pure-efi.fd , file however.

I don't see a way to turn VGA off in the GUI. Is there an option that is preferable to use, and does not place a virtual VGA card on the vm?

Lastly, I wonder if you know why PM stops me from migrating the VM. If I have identical hardware on the other side, why not? I understand it is more stuff to maintain for a niche application, however.

I wasn't being hyperbolic above when I said it was 'like magic'. I have to say, it is not often technology impresses me anymore, but this is pretty cool.
 
Last edited:
spirit said:
That's great it's working out of the box with ovmf.
Click to expand...

I wanna correct something I said above for posterity: Aside from the hostpci: stuff, I also had to add "machine: q35" to vmid.conf file. The wiki is somewhat ambiguous as to what is and is not necessary in the example it gives for UEFI GPU passthrough.

spirit said:
BTW, if you add ",x-vga=on" to the hostpci, it should do the tricks for hv_vendor, vga=none, etc...
Click to expand...

Due to the current bug whereby OVMF bioses do not save, I was unable to get to a desktop until I passed some ports for USB HID. I was able to test it for the first time today with "x-vga=on", and as soon as I got to the desktop I started seeing problems in the form of scrambled video and display driver crashes, usually before I even got to a desktop.

Unfortunately, it seems that "hv_vendor_id=proxmox" -- the option that is set when "x-vga=on" is specified -- is probably not correct in all cases. Specifically, it may be a problem for all nvidia cards.

I was able to get everything working smoothly by changing that -cpu option manually to "hv_vendor_id=Nvidia43FIX".

The only other way I was able to find was to remove all the hv_stuff, and leave only kvm=off, which is probably not preferable, I would assume.

It would be nice to be able to start and stop the VM from the GUI. I assume it would be a pretty easy temporary change to make to the script that generates the kvm command. Could you point me in the right direction?

Lastly, I just have to say again that this is really cool.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back