[SOLVED] Can't make Nvidia GPU passthrough to WS2019 guest on HP ML110 Gen9

[ukro]

Greetings,
i tried official proxmox wiki, all reddit posts, and 2 pages of google links and no luck, Please help!

Virtual Environment 7.3-3 (fully updated today free repository)

Sometimes guest showing the GPU GT710 with 43 error and sometimes not showing at all, sometimes black screen when installing drivers (it was during the whole time that i tested different settings on host). I also tried taking out the ROM and using it in vm.conf no luck. Tried with geforce experience, without, snappy driver kit...

Actual settings shows windows basic drivers with the PCI slot of the external GPU, when installing drivers black screen apears, no network connection as if it crash, after putting the vga settings to se the display output,not booting, trying CDROM and after freezing proxmox logo.

nano /etc/pve/qemu-server/101.conf

bios: ovmf
boot: order=ide0;virtio0
cores: 4
efidisk0: vms:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00,pcie=1,x-vga=1
ide0: local:iso/en_windows_server_2019_vl_x64_by_AG_01.2022.iso,media=cdrom,size=5252544K
ide2: local:iso/virtio-win-0.1.225.iso,media=cdrom,size=519590K
machine: q35
memory: 8192
meta: creation-qemu=7.1.0,ctime=1673879060
name: win2019
net0: virtio=36:28:60:2A:65:11,bridge=vmbr0,firewall=1
net1: virtio=B2:06:80:F9:F5:83,bridge=vmbr2,firewall=1
numa: 0
ostype: win10
parent: beforedriverinstall
scsihw: virtio-scsi-pci
smbios1: uuid=7dd84fb4-0002-44e6-b5a6-45bdfdbb27fe
sockets: 1
vga: none
virtio0: vms:vm-101-disk-1,size=100G
vmgenid: d29857fe-affe-4b45-ad33-7bae544bf02c

grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off,vesafb:off"

cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1 report_ignored_msrs=0

cat /etc/modprobe.d/blacklist.conf

blacklist nvidiafb
blacklist radeon
blacklist nouveau
blacklist nvidia

cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:128b,10de:0e0f disable_vga=1

cat /etc/modprobe.d/iommu_unsafe_interrupts.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1 report_ignored_msrs=0

dmesg | grep 'remapping'

[    0.237720] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.238299] DMAR-IR: Enabled IRQ remapping in x2apic mode

dmesg | grep -i vfio

[    5.241080] VFIO - User Level meta-driver version: 0.3
[    5.249874] vfio-pci 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    5.270220] vfio_pci: add [10de:128b[ffffffff:ffffffff]] class 0x000000/00000000
[    5.293774] vfio_pci: add [10de:0e0f[ffffffff:ffffffff]] class 0x000000/00000000

/etc/modules

coretemp
ipmi_devintf
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

IOMMU

IOMMU Group 57:
        03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
        03:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)

 

[leesteken]

Just an observation:

grub:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off,vesafb:off"

video=efifb:off,vesafb:off is ignored. You need to write it as video=efifb:off video=vesafb:off. But that does nothing since kernel 5.15 and you need video=simplefb:off. But that does not work for single (or boot) GPU passthrough: you need this work-around.

 

[Zetto]

Just an observation:

video=efifb:off,vesafb:off is ignored. You need to write it as video=efifb:off video=vesafb:off. But that does nothing since kernel 5.15 and you need video=simplefb:off. But that does not work for single (or boot) GPU passthrough: you need this work-around.

You Legend.....I'm tinkering around with this for the last 3 days and nothing working until this.......Thank you very much. (I hope it solves it for you too Ukro)

 

[ukro]

Just an observation:

video=efifb:off,vesafb:off is ignored. You need to write it as video=efifb:off video=vesafb:off. But that does nothing since kernel 5.15 and you need video=simplefb:off. But that does not work for single (or boot) GPU passthrough: you need this work-around.

Thank you for pointers.
I left all the settings as is and just changed the grub. I have tried with Seabios and OMVF.
I tried with romfile and without,x-vga and without

OMVF
When installing with installer - blank screen and boot problem
When installing only drivers manualy - blank screen and boot problem
the card didn't even apear, only the default windows monitor

Seabios
the card shows correctly in device manager but with error 43

 

[ukro]

You Legend.....I'm tinkering around with this for the last 3 days and nothing working until this.......Thank you very much. (I hope it solves it for you too Ukro)

Can you please be so kind and write all the settings that you have in the files that i have in a posted? I might have some small changes that you don't have and its still not working for me.

 

[leesteken]

Thank you for pointers.
I left all the settings as is and just changed the grub. I have tried with Seabios and OMVF.
I tried with romfile and without,x-vga and without

OMVF
When installing with installer - blank screen and boot problem
When installing only drivers manualy - blank screen and boot problem
the card didn't even apear, only the default windows monitor

Seabios
the card shows correctly in device manager but with error 43

Maybe the GT710 does not have UEFI firmware? Sometimes it is necessary for NVidia GPUs to get the firmware ROM from the GPU (by booting with another GPU) and patch it, and pass that ROM-File in the passthrough settings.
Maybe the NVidia drivers have issues with the QEMU 7.1 machine model. Try using version 6.2, but that might require reinstalling Windows, I don't know.
Disable Above 4G Decoding and Resizable BAR in your system BIOS.

 

[ukro]

Maybe the GT710 does not have UEFI firmware? Sometimes it is necessary for NVidia GPUs to get the firmware ROM from the GPU (by booting with another GPU) and patch it, and pass that ROM-File in the passthrough settings.
Maybe the NVidia drivers have issues with the QEMU 7.1 machine model. Try using version 6.2, but that might require reinstalling Windows, I don't know.
Disable Above 4G Decoding and Resizable BAR in your system BIOS.

I extracted the ROM file and it was stated type:3 from which i have read that what i need.
Why you are saing Nvidia drivers? I didn't install any drivers on proxmox. The drivers are on guest VM.
I am not going back to 6.2 D: maybe sometime for testing. Or maybe i will try another GPU, maybe amd?
I don't remember seeing any resizable BAR or 4G decoding in HP ML110 Gen9 bios, will go through it again.
Thank you for pointers <3

 

[leesteken]

I extracted the ROM file and it was stated type:3 from which i have read that what i need.

I have no experience with that, sorry. You are not using the ROM-File according to the VM configuration file. Maybe try that?

Why you are saing Nvidia drivers? I didn't install any drivers on proxmox. The drivers are on guest VM.

I know, and I do mean the NVidia drivers inside the VM. Maybe they give a code 43 because of the QEMU machine version being too new/different.

I am not going back to 6.2 D: maybe sometime for testing. Or maybe i will try another GPU, maybe amd?

This look to me like you think I meant Proxmox 6.2. Please try setting the QEMU machine version of your current VM to 6.2 (in the Proxmox GUI go to VM settings, Hardware, Machine, change Version) and see if that makes things better for Windows and NVidia drivers inside the VM.

I don't remember seeing any resizable BAR or 4G decoding in HP ML110 Gen9 bios, will go through it again.

I have no experience with HP and recent Intel, so I cannot help you find them. Modern BIOSes should Above 4G Decoding (usually under Boot Options) but maybe yours does not have Resizable BAR yet.

 

[ukro]

This look to me like you think I meant Proxmox 6.2. Please try setting the QEMU machine version of your current VM to 6.2 (in the Proxmox GUI go to VM settings, Hardware, Machine, change Version) and see if that makes things better for Windows and NVidia drivers inside the VM.

Sorry, tried with 5.1, 6.0, 7.1 same result.
I was playing with the qemu machine version and once the error 43 disappear i was so happy but after reboot it appears again
But i have snapshoted windows where it is without error,so i can go back....but still after reboot it have 43error..
P.S. and i am doing it on seabios as on OMVF it is blank screen all the time after driver update

 

[leesteken]

Sorry, tried with 5.1, 6.0, 7.1 same result.

Everything except the 6.2 that i suggested? ;-)

I was playing with the qemu machine version and once the error 43 disappear i was so happy but after reboot it appears again

Reboot of the VM or the host? Some devices just don't reset properly and only work once per host reboot.

But i have snapshoted windows where it is without error,so i can go back....but still after reboot it have 43error..
P.S. and i am doing it on seabios as on OMVF it is blank screen all the time after driver update

Sorry but I have no experience with NVidia drivers but maybe someone else knows the trick here.

 

[ukro]

Everything except the 6.2 that i suggested? ;-)

Reboot of the VM or the host? Some devices just don't reset properly and only work once per host reboot.

Sorry but I have no experience with NVidia drivers but maybe someone else knows the trick here.

Sorry 6.2 :-X

Everything except the 6.2 that i suggested? ;-)

Reboot of the VM or the host? Some devices just don't reset properly and only work once per host reboot.

Sorry but I have no experience with NVidia drivers but maybe someone else knows the trick here.

VM

Everything except the 6.2 that i suggested? ;-)

Reboot of the VM or the host? Some devices just don't reset properly and only work once per host reboot.

Sorry but I have no experience with NVidia drivers but maybe someone else knows the trick here.

<3 thank you i will continue to poke the qemu machine further ))

 

[Zetto]

Can you please be so kind and write all the settings that you have in the files that i have in a posted? I might have some small changes that you don't have and its still not working for me.

Your settings look fine to me but we got quite different systems. I'm running it on a consumer-grade motherboard with i7 8700 and GPU is RTX2060. While you're on server hardware with Xeon's I presume. I saw some posts that Xeon's are a bit finicky when it comes to IOMMU. I also remember seeing a post that said that it is harder to virtualize older GPU's but I can't find it anymore.......I've read a lot about this in the last 3 days.

 

[ukro]

Your settings look fine to me but we got quite different systems. I'm running it on a consumer-grade motherboard with i7 8700 and GPU is RTX2060. While you're on server hardware with Xeon's I presume. I saw some posts that Xeon's are a bit finicky when it comes to IOMMU. I also remember seeing a post that said that it is harder to virtualize older GPU's but I can't find it anymore.......I've read a lot about this in the last 3 days.

Got it, thank you for info

 

[ukro]

So i was able to remove 43error and to get an output signal from external GPU, i can see windows screen, it but after installing nvidia drivers display go black and windows not booting. On boot i see Proxmox splashscreen and then stuck.

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' -machine 'max-ram-below-4g=1G,kernel_irqchip=on'
bios: ovmf
boot: order=virtio0;ide2;sata0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: vms:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00,pcie=1,x-vga=1
machine: pc-q35-5.2
memory: 8128
meta: creation-qemu=7.1.0,ctime=1673976195
name: win10
net0: virtio=3E:A9:68:F3:78:28,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: drivers_deleted_booting_nonedisplay
scsihw: virtio-scsi-single
smbios1: uuid=079631b6-1c98-4e2d-ae5a-287a0845cfd3
sockets: 1
vga: none
virtio0: vms:vm-102-disk-1,iothread=1,size=40G
vmgenid: 0938c796-4297-4d3e-9556-a4151785e85a

Any suggestions?

 

[ukro]

I gave up... One suggestion that I read that try another GPU....newer...$$$$$$

 

[ukro]

Done but went for Windows 10, unfortunately same settings didn't worked on WS2019 which is odd.
Windows 10 with GT710
Nvidia drivers 474.14
Proxmox Virtual Environment 7.3-3

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' -machine 'max-ram-below-4g=1G,kernel_irqchip=on'
bios: ovmf
boot: order=virtio0;ide2;sata0
cores: 4
cpu: host,hidden=1,flags=+pcid
efidisk0: vms:vm-102-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:03:00,pcie=1,romfile=gt7101gnew.bin
machine: pc-q35-6.2
memory: 8128
meta: creation-qemu=7.1.0,ctime=1673976195
name: win10
net0: virtio=3E:A9:68:F3:78:28,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: drv_installed_regedited_gmailukrospm
sata0: local:iso/en_windows_server_2019_vl_x64_by_AG_01.2022.iso,media=cdrom,size=5252544K
scsihw: virtio-scsi-single
smbios1: uuid=079631b6-1c98-4e2d-ae5a-287a0845cfd3
sockets: 1
virtio0: vms:vm-102-disk-1,iothread=1,size=40G
vmgenid: 94080bc5-73ea-4b85-9aed-e2ef4eb9b4b9

Took bios online as the exported was broken:
https://www.techpowerup.com/vgabios/236505/236505
Patched the BIOS with this:
https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher
Done this registry edit multiple times first with pcie in Hardware, then without, then with pcie again:
https://forums.engineerworkshop.com/t/fixing-gpu-passthrough-on-windows-10-vm-in-proxmox/207
And this would be my last option but didn't needed it:
https://forum.proxmox.com/threads/gpu-passthrough-code-43-unable-to-resolve.107996/post-506771