Is anybody who succeeded GPU passthrough (NVIDIA) with Windows 7 ?

[Stargazer]

Thank you for enduring my bad English, first. ^ _^ ;

I used KVM with libvirt (virt-manager) and 'migrated' to Proxmox recently.

I had succeeded GPU passthrough in libvirt with win7 win8.1 win10 but now I just do it with win10 with Proxmox.
(win 10 seems very easy as it requires 'almost' only GPU rom)

Win 8.1 seems to be easy but it isn't because it looks succeeded but the screen display isn't
although it shows menu "Nvidia Control Panel" and 3d accelated Chrome doesn't show up (looks white blank screen.

Win 7 is stuck on boot ( I suppose with cpu usage because I can't look at without VNC)
My settings is


....
machine: q35
hostpci0: 01:00.0;01:00.1, pcie=1, x-vga=on, romfile=1.rom
.....

- pre-required works are done (iommu, vfio, acs patch, rom file ......)
- installed Win 7 as UEFI with OVMF
- boot option edited with boot up screen and set only "PCI-E Device" boot
- manually updated kernel and qemu-kvm
- added efi disk

I set almost similar circumtances with similar options as libvirt with virt-manager,
but just stuck on boot in Proxmox
Is there any hints?

Thanks in advance.

 

[frickelpit]

I have the same problem. Win7 stucks on the bootscreen logo. With RDP connect you can see, the grafic card becomes a code 12 error in the device manager. No matter if UEFI or seabios. The result is the same. I think, it's actually not possible because nobody has a running config with gpu passthrough and proxmox 5.2.

 

[dcsapak]

mhmm, could you post the output of 'qm showcmd ID' ?
maybe some cpu/hv flags are not good for windows 7

 

[frickelpit]

No problem...

Spoiler: qm showcmd 205

/usr/bin/kvm -id 205 -chardev 'socket,id=qmp,path=/var/run/qemu-server/205.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/205.pid -daemonize -smbios 'type=1,uuid=d4c04550-2da2-447e-8d4c-a2b38f192ed0' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-205-disk-2' -name WS5 -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,kvm=off' -m 4096 -k de -object 'iothread,id=iothread-virtio1' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'usb-host,vendorid=0x05ac,productid=0x0221,id=usb0' -device 'usb-host,vendorid=0x045e,productid=0x00cb,id=usb1' -device 'usb-host,vendorid=0x04b8,productid=0x011f,id=usb2' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:17d28020a41f' -drive 'file=/pool1/TEMPLATES/template/iso/virtio-win-0.1.141.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2' -drive 'file=/dev/pve/vm-205-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -drive 'file=/dev/zvol/pool1/vm-205-disk-1,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1' -netdev 'type=tap,id=net0,ifname=tap205i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B2:6E:2F:28:3D:8A,netdev=net0,bus=pci.0,addr=0x12,id=net0' -machine 'type=q35'

 

[dcsapak]

can you try to set the windows version to >= win8 (even though its running win7)?

 

[frickelpit]

With "win8" in config win7 crashes on boot logo screen.

 

[Stargazer]

In libvirt/virt-manager, Win7 was successful because I can directly assign PCI-E device to boot device.

I finally know it's unable in Proxmox because I can't assign GPU to boot.

Windows 8.1, as I said above, it seems work normally but it doesn't show 3D part of screen.
(as I know till now)

 

[JustAMan]

Hello there!

I've found the relevant bug against qemu-kvm,. google for "Red Hat Bugzilla – Bug 1273172" (I cannot post direct link to this bug as I'm a new user and not allowed to post external links so far).

Following the ideas in that bug I was able to bring up a Win7 VM with NVidia GTX 1080 Ti passed through (last night Unigine Heaven on 4K output with everything on max yielded a score of about 970, while I see native results ranging from 980 to 1010, so not that bad).

The problem haunting my configuration (error code 12 - not enough resources) has something to do with the fact that by default host PCI devices are passed through as though they are behind a PCI bridge (which works just fine on Win10 and MacOS High Sierra, BTW, on the same hardware).

Below is the configuration file for my VM. I have passed through VGA, HDMI audio and a USB controller.

vga: none
agent: 1
args: -cpu host,hv_time,kvm=off,hv_vendor_id=TheNeverhood,-hypervisor -device vfio-pci,host=02:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on -device vfio-pci,host=02:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1
balloon: 0
bios: ovmf
boot: dcn
bootdisk: scsi1
cores: 4
cpu: host
efidisk0: local:101/vm-101-disk-2.qcow2,size=128K
hostpci1: 00:14.0,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 8192
name: win7main
net0: virtio=E2:68:03:36:43:6A,bridge=vmbr0
numa: 0
ostype: win7
scsi1: local:101/vm-101-disk-1.qcow2,discard=on,iothread=1,size=120G
scsihw: virtio-scsi-pci
smbios1: uuid=4c2c2891-99b8-4287-b057-9f2aeaad7692
sockets: 1

Hope that helps other people interested in such a thing as Windows 7 with passed through video card...

 

[frickelpit]

Best hint ever! I have tried so many hours to run this shitty W7 with passed through video card. I always run into error 12. Now i can use my old CAD soft virtualised. GREAT!!!

 

[Proxmox India]

did this finally work ?

 

[frickelpit]

Yes. It's not a joke!

 

[frickelpit]

After testing a while i have to say unfortunately it works only one time. After shutting down the win7 vm, it is not possible to start again or to start another vm for example win10 with gpu passthrough. The host system freeze shortly after vm start

I think that the proxmox mechanism to release the grafic card isn't made for that use case.... ?

 

[dcsapak]

I think that the proxmox mechanism to release the grafic card isn't made for that use case.... ?

unfortunately this is more related to the hardware and the guest os (which is responsible to release the devices)

 

[JustAMan]

My own setup (ASUS Z170 PRO motherboard + NVidia 1080 Ti video) works fine so far.
I did not give it any significant load, though, just ran Unigine Heaven a couple of times.

What video card do you have?

 

[frickelpit]

I use a Fujitsu DS D3417-B2 Mobo with a XEON E3-1245 v6 and 64GB ECC-RAM. Grafic card is a NVidia Quadro P2000 with driver v391.89 WHQL for Win7x64.

 

[JustAMan]

I have tinkered my own version of qemu-server that is able to control how to pass the card (as it seems it does a few more things when you put "x-vga=on" in the hostpci line compared to my initial config I posted above).
Let me get back to home and I can try and show you qemu command line so we can try to compare mine and yours and see if there's some difference.

Then you can try to construct a commandline yourself and see if patched commandline fixes your issues.

 

[JustAMan]

Here is the result of qm showcmd 101 (101 is the id of my Win7 VM):

/usr/bin/kvm -id 101 -name win7main -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=4c2c2891-99b8-4287-b057-9f2aeaad7692' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=qcow2,id=drive-efidisk0,file=/var/lib/vz/images/101/vm-101-disk-2.qcow2' -smp '6,sockets=1,cores=6,maxcpus=6' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'Skylake-Client,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,kvm=off,vendor=GenuineIntel' -m 8192 -object 'iothread,id=iothread-virtioscsi1' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on' -device 'vfio-pci,host=02:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1' -device 'vfio-pci,host=00:14.0,id=hostpci1,bus=ich9-pcie-port-2,addr=0x0' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:cd1a3fd61c95' -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -device 'virtio-scsi-pci,id=virtioscsi1,bus=pci.3,addr=0x2,iothread=iothread-virtioscsi1' -drive 'file=/var/lib/vz/images/101/vm-101-disk-1.qcow2,if=none,id=drive-scsi1,discard=on,format=qcow2,cache=none,aio=native,detect-zeroes=unmap' -device 'scsi-hd,bus=virtioscsi1.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi1,id=scsi1,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=E2:68:03:36:43:6A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'

Try comparing with yours to see if there's any difference.

 

[frickelpit]

Thx. I have tested it, but nothing helps. Now my vm.cfg looks like yours. But with the difference that my second card which i passthrough to the vm is a pci (USB3-Card) and not a pcie card. I have tried to add the device-line from the USB3-Card to the args-line in cfg, but Proxmox find this not usefull

If i run this command at console proxmox can't find "bus=pci.0", if i start same cfg with webgui the vm starts without a problem?!

Spoiler: conf.cfg

/usr/bin/kvm -id 203 -name CAD7 -chardev 'socket,id=qmp,path=/var/run/qemu-server/203.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/203.pid -daemonize -smbios 'type=1,uuid=a7d9be4d-014c-476f-b42f-ab7bc1738471' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-203-disk-2' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'Skylake-Client,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,kvm=off,vendor=GenuineIntel' -m 8192 -object 'iothread,id=iothread-virtio1' -readconfig /usr/share/qemu-server/pve-q35.cfg  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on' -device 'vfio-pci,host=01:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1' -device 'vfio-pci,host=02:00.0,id=hostpci1,bus=pci.0,addr=0x11' -chardev 'socket,path=/var/run/qemu-server/203.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:17d28020a41f' -drive 'file=/dev/pve/vm-203-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -drive 'file=/dev/zvol/pool1/vm-203-disk-1,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1' -netdev 'type=tap,id=net0,ifname=tap203i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B2:6E:2F:28:3D:8A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'

btw... the device manager in win7 is free of errors or warnings. The problem comes after the shutdown from win7. Start another vm and the host will be frozen.

 

[JustAMan]

What happens if you start this VM again after shutting it down, does it work? I can not test other VMs now as I don't have them yet.
Also, what is the output when you double-click "Start" task in task log? And is there anything interesting in dmesg after turning problematic VM on or off?

 

[frickelpit]

Problem solved!
The reason was the USB3 PCI-Card which i want to passthrough to the Win-VMs. At host start the card becomes the kernel driver "xhci_hcd", same as the host use for his USB3-chip. WIN10 has no problems with this, differently to WIN7. I haven't found a way to prevent the card from loading the driver because it's a kernel driver and not a modul.

So i passthrough the usb ports and not the hole pci card. Not nice but it works