[SOLVED] GPU Passthrough (windows problems)

[floh]

Hello guys!

I'm currently trying to passthrough a NVIDIA P4000 into a Windows 10 VM.

I've already read the WIKI article and checked this forum as well as some reddit threads - THIS thread is (imho) really helpfull.

The Problem I'm currently facing is that I can passthrough the gpu but windows says that there is a problem and so windows disables the gpu (drivers are installed and windows "knows" already that the GPU is a NVIDIA P4000).
I tried to use the GPU as "Primary GPU" but then starting the VM doesn't work:

kvm: -device vfio-pci,host=0000:12:00.0,id=hostpci0,bus=pci.0,addr=0x10,x-vga=on: vfio 0000:12:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga
TASK ERROR: start failed: QEMU exited with code 1

Without x-vga (not as primary gpu) the VM is able to start but it also displays the following:

audio: Could not init `spice' audio driver
audio: warning: Using timer based audio emulation
kvm: vfio: Cannot reset device 0000:12:00.0, depends on group 44 which is not owned.
kvm: vfio: Cannot reset device 0000:12:00.0, depends on group 44 which is not owned.
TASK OK

One last thing to mention:
When I type dmesg | grep -e DMAR -e IOMMU in the nodes shell I do not get any output (wiki says that there should be any output otherwise something else is broken)


Does anybody know how to fix this?

best regards,
Floh

 

[Stefan_R]

Can you post your VM config? (I.e. /etc/pve/qemu-server/<vmid>.conf) Also the output of find /sys/kernel/iommu_groups/ -type l and lspci -nn.

One last thing to mention:
When I type dmesg | grep -e DMAR -e IOMMU in the nodes shell I do not get any output (wiki says that there should be any output otherwise something else is broken)

Yes, there should be. This usually indicates that you didn't set up your IOMMU (i.e. VT-d) correctly, but considering you already have the pass-through partially working that seems to be incorrect... What does cat /proc/cmdline say?

Have you tried passing through other devices than the GPU to see if it's a problem with the graphics card or passthrough in general? For example try passing through a network card or a USB controller, even the onboard ones often work.

 

[floh]

Hello Stefan!

Thank you for your quick response. I'll provide all necsessay data - hope it'll help.

Spoiler: vm-config

root@[...nodename...]:~# cat /etc/pve/qemu-server/349.conf
#Last Update - 09.04.2020
agent: 1
args: -device intel-hda,id=sound5,bus=pci.0,addr=0x18 -device hda-micro,id=sound5-codec0,bus=sound5.0,cad=0 -device hda-duplex,id=sound5-codec1,bus=sound5.0,cad=1
audio0: device=intel-hda,driver=spice
balloon: 4096
bootdisk: virtio0
cores: 20
efidisk0: Ceph-Storage:vm-349-disk-2,size=1M
hostpci0: 12:00.0
ide2: Ceph-ISO-Storage:iso/virtio-win-0.1.171.iso,media=cdrom,size=363020K
memory: 262144
name: vmname
net0: virtio=BA:A1:85:2A:C6:AA,bridge=vmbr0,firewall=1,link_down=1
net1: e1000=FA:32:0B:57:11:5C,bridge=vmbr20,firewall=1
net2: virtio=82:DC:56:E6:53:C2,bridge=vmbr10,firewall=1
net6: virtio=8E:22:80:2F:B8:B5,bridge=vmbr70,firewall=1
numa: 0
ostype: win10
scsi0: Ceph-Storage:vm-349-disk-1,discard=on,size=512G
scsihw: virtio-scsi-pci
smbios1: uuid=d4369df5-f243-42e0-8f85-0eeda7900343
sockets: 1
virtio0: Ceph-Storage:vm-349-disk-0,discard=on,size=64G
vmgenid: c2822f97-d98a-471d-ac3b-e70b13e75526

root@[...nodename...]:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.3.13-3-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off

When starting the VM with a USB-Controller passed through I get the same start-message:

audio: Could not init `spice' audio driver
audio: warning: Using timer based audio emulation
kvm: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.
kvm: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.
TASK OK

But it seams to work fine.

- Floh

 

[Stefan_R]

Have you enabled "All Functions" in the GUI? The config you posted seems to be for the variant with the USB controller..

I don't see anything strikingly wrong with your config though... Potentially the pcie_acs_override? Do you actually need that? It's more of a hack than a real fix, so I could imagine this breaking things.

 

[floh]

I've tried it with All Functions enabled, also with disabled.

The config is with the nvidia gui ... "hostpci0: 12:00.0" --> 12.:00 should be the nvidia graphics card.

I don't see anything strikingly wrong with your config though... Potentially the pcie_acs_override? Do you actually need that? It's more of a hack than a real fix, so I could imagine this breaking things.

Ohh I've added this parameter because it wasn't working without either - I'll remove it just now for a cleaner config.

 

[floh]

Good Morning!

I just started the VM again with "All Functions" - this time the "start-log" which is under "Tasks" is pretty empty:

audio: Could not init `spice' audio driver
audio: warning: Using timer based audio emulation
TASK OK

But windows stops still the gpu because of "problems". A screenshot of the Device Manager is attached.

- Floh

 

[sirkope]

Hello,

If I know well, spice audio won't work together with GPU passtrough.
Display settings should be none. But before that, under Windows guest you should enable Remote Desktop (under Linux you should install one, for me, XRDP seems working.) Before you change Display setting to none, try the Remote Desktop.
Code 43, well, its harder, I'm still suffering from it, rumor says, NVIDIA driver recognise virtualisation, and won't work.
There is a possible solution: https://github.com/sk1080/nvidia-kvm-patcher ,but I couldn't manage it (yet) I"ve tried it on the win10 guest, but at the weekend, I will try on a 'real' hardver. But at least I have sound.

 

[floh]

Hello!

Thanks you for the hint to change display settings to none. Sadly I've tried it already getting the same error message (code 43) - so I enabled it again because of comfort reasons (the pve-console is really nice and easy/fast to use).

Aside from the bad news above I have really good news - game changing for me:
I've read in a qemu related thread that some guestsystems seem to work better with PCI-express cards than with PCI devices. So I tried to enable the pci-e flag. (Which does not work out of the box - TASK ERROR: q35 machine model is not enabled at /usr/share/perl5/PVE/QemuServer.pm line 3585.)

To resolve the error mentioned above you have to change in the VMs hardware the "Machine"-option to "q35".
Then start the VM again. (Hint: if you get some error like "device does not support requested feature x-vga, disable the flag "Primary GPU")

Spoiler: Here the complete error-message '[...] does not support [...] x-vgs'

kvm: -device vfio-pci,host=0000:12:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0,x-vga=on: vfio 0000:12:00.0: failed getting region info for VGA region index 8: Invalid argument
device does not support requested feature x-vga
TASK ERROR: start failed: QEMU exited with code 1

surprise: It worked - windows recognize the gpu without any problems (with Display-setting="Default", so the proxmox-console-tab works too).
Windows now has two GPUs in the device-list but from there on you can google it yourself how to start applications with a specific gpu ;-)


Hopefully this thread can help others as well.

- Floh

 

[sirkope]

I'm glad you've made it.

Could you send me the output of this command from pve shell?

qm showcmd <vmid> --pretty

for example: qm showcmd 104 --pretty

 

[floh]

Yes of course - here it is:

root@[...nodename...]:~# qm showcmd 349 --pretty
/usr/bin/kvm \
  -id 349 \
  -name [...name...] \
  -chardev 'socket,id=qmp,path=/var/run/qemu-server/349.qmp,server,nowait' \
  -mon 'chardev=qmp,mode=control' \
  -chardev 'socket,id=qmp-event,path=/var/run/qmeventd.sock,reconnect=5' \
  -mon 'chardev=qmp-event,mode=control' \
  -pidfile /var/run/qemu-server/349.pid \
  -daemonize \
  -smbios 'type=1,uuid=d4369df5-f243-42e0-8f85-0eeda7900343' \
  -smp '20,sockets=1,cores=20,maxcpus=20' \
  -nodefaults \
  -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' \
  -vnc unix:/var/run/qemu-server/349.vnc,password \
  -no-hpet \
  -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,hv_synic,hv_stimer,hv_ipi,enforce' \
  -m 262144 \
  -device 'vmgenid,guid=c2822f97-d98a-471d-ac3b-e70b13e75526' \
  -readconfig /usr/share/qemu-server/pve-q35-4.0.cfg \
  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' \
  -device 'vfio-pci,host=0000:12:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,rombar=0' \
  -device 'intel-hda,id=audiodev0,bus=pci.2,addr=0xc' \
  -device 'hda-micro,id=audiodev0-codec0,bus=audiodev0.0,cad=0' \
  -device 'hda-duplex,id=audiodev0-codec1,bus=audiodev0.0,cad=1' \
  -audiodev 'spice,id=spice-backend0' \
  -device 'VGA,id=vga,bus=pcie.0,addr=0x1' \
  -chardev 'socket,path=/var/run/qemu-server/349.qga,server,nowait,id=qga0' \
  -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' \
  -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' \
  -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' \
  -iscsi 'initiator-name=iqn.1993-08.org.debian:01:c3fe7cf18e64' \
  -drive 'if=none,id=drive-ide2,media=cdrom,aio=threads' \
  -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' \
  -device 'virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5' \
  -drive 'file=rbd:Ceph-Storage/vm-349-disk-1:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/Ceph-Storage.keyring,if=none,id=drive-scsi0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
  -device 'scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0' \
  -drive 'file=rbd:Ceph-Storage/vm-349-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/Ceph-Storage.keyring,if=none,id=drive-virtio0,discard=on,format=raw,cache=none,aio=native,detect-zeroes=unmap' \
  -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=101' \
  -netdev 'type=tap,id=net0,ifname=tap349i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
  -device 'virtio-net-pci,mac=BA:A1:85:2A:C6:AA,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' \
  -netdev 'type=tap,id=net1,ifname=tap349i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' \
  -device 'e1000,mac=FA:32:0B:57:11:5C,netdev=net1,bus=pci.0,addr=0x13,id=net1,bootindex=301' \
  -netdev 'type=tap,id=net2,ifname=tap349i2,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
  -device 'virtio-net-pci,mac=82:DC:56:E6:53:C2,netdev=net2,bus=pci.0,addr=0x14,id=net2,bootindex=302' \
  -netdev 'type=tap,id=net6,ifname=tap349i6,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' \
  -device 'virtio-net-pci,mac=8E:22:80:2F:B8:B5,netdev=net6,bus=pci.1,addr=0x1,id=net6,bootindex=303' \
  -rtc 'driftfix=slew,base=localtime' \
  -machine 'type=q35+pve1' \
  -global 'kvm-pit.lost_tick_policy=discard' \
  -device 'intel-hda,id=sound5,bus=pci.0,addr=0x18' \
  -device 'hda-micro,id=sound5-codec0,bus=sound5.0,cad=0' \
  -device 'hda-duplex,id=sound5-codec1,bus=sound5.0,cad=1'

 

[sirkope]

Thx, I'll compare it with mine.