[SOLVED]vfio-nv43

[midgetseeker]

Hello everyone,

I'm trying to move away from vmware esxi 7 but it's hard.
I've done a dump of the vbios and patched it.
That seems to work with the windows installer just fine up until you add a network card and windows pulls the nvidia stock driver and it goes sideways.

How exactly one get gpu passthrough working on proxmox???

I have dumped the vbios with gpu-z on a baremetal w10 install. Patched it with the vfio patcher. This is an 1080ti gp102 card.
Why is it so hard to get gpu passthrough working?

I've attached my config.

9000 is a template of win10
101 is a deploy of that template

 

[midgetseeker]

Any help is appreciated. Definitely it's a small thing but then again there are a LOT of instructions, that don't seem to work.

 

[midgetseeker]

Bumping this thread hoping someone take pity on me and maybe points out what I'm doing wrong.

 

[leesteken]

Instead of pc-q35-7.1, use pc-q35-6.2, which helps a lot with AMD Windows 10 drivers. You already have Primary GPU (x-vga=1) and romfile.
If you are passing through the boot GPU, use this work-around, instead of nofb nomodeset video=vesafb:off video=efifb:off video=simplefb:off which does nothing since Proxmox 7.2.
Using that work-around of initcall_blacklist=sysfb_init usually solves the BAR 3: can't reserve error, as you can see from that thread. I did not recognize your issue earlier because the error message was hidden in the many attachments and you did not describe your hardware setup, sorry,

EDIT: You did provide all the necessary information, which is excellent, but I did not recognize it before. Again, I offer my apologies.

 

[midgetseeker]

Thank you @leesteken for your reply,

I will try to lower the VM HW level and set that kernel param.

Bare in mind, to avoid looking at too many files, that this is an Intel platform.

It's an Asus x299 Sage WS board with an Intel 9800X CPU.
The bios is really troublesome on these boards as they are meant for CAD/Video editing rather than what I want which is to hyper converge use in the form of running VMs for testing/developing and also play games.

I've managed to do that with VMware ESXi 7.0.3 and vCenter but the requirements to run this setup are affected by electricity prices and my budget.
Proxmox provides a lot of flexibility as I'm also running it on some raspberryPIs and would love to have all of them in a cluster but I digress

I'll test and come back with results and some relevant debug data.
Thank you so much for the hint!

 

[midgetseeker]

Hey @leesteken

I am humbled, spent 2 weeks trying various settings but this indeed made it work!

Can't express myself how happy I am with how this worked. I did probably 90% of what was needed to make it work since I was able to install win10 in the VM from the Proxmox host itself but was always stuck at code 43 driver error whenever the microsoft supplied update driver or the official nVidia driver got installed.

Just did the setting, created a new VM and a fresh install of win10 just to make sure I'm not hitting a problem created from my previous attempts, the nVidia driver installed just like it would on bare metal.

I will document further with details but definitely google is NOT my friend.
Just to proof test I quickly fired up Doom Eternal to see if I get a crash and it worked as usual.

Again, thank you so much for your help!

 

[midgetseeker]

Hello everyone,

I'll post my configuration details for anyone in my situation to reference. Mileage may vary with regards to HOST/Video board manufacturer.

I'll just cover the Proxmox configuration state as my issues were caused by the amount of information that is deprecated or just false/speculation/works for me.

root@px-desktop:~# uname -a
Linux px-desktop 5.15.74-1-pve #1 SMP PVE 5.15.74-1 (Mon, 14 Nov 2022 20:17:15 +0100) x86_64 GNU/Linux

root@px-desktop:~# pveversion
pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve)

root@px-desktop:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-5.15.74-1-pve root=/dev/mapper/pve-root ro quiet intel_iommu=on initcall_blacklist=sysfb_init

root@px-desktop:~# cat /etc/modprobe.d/blacklist.conf
blacklist radeon
blacklist nouveau
blacklist nvidia

root@px-desktop:~# cat /etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

root@px-desktop:~# cat /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0

root@px-desktop:~# cat /etc/modprobe.d/pve-blacklist.conf
# This file contains a list of modules which are not supported by Proxmox VE

# nidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb

root@px-desktop:~# cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1b06,10de:10ef disable_vga=1

The IOMMU group looks like this:

IOMMU Group 76:
    68:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
    68:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)

One thing to note is that the Audio controller needs to be in the same IOMMU group with the Video Controller. If they are not something is wrong.

Before I got it working with the above solution the Video and Audio components of the card were showing in IOMMU 76 & 77 respectively.

The VFIO Patch is NOT necesary for this to work. The only difference between using the card's vbios dump or a patched dump is just the UEFI start screen when the machine boots.

A patched vbios dump shows the UEFI splash screen when the VM starts whereas the original vbios will initialize display when the OS does it.
THIS IS APPLICABLE TO A 1080Ti NVIDIA CARD!

Here's my VM config. Device at hostpci0 is an USB 3.2 adapter

#Fresh install of w10, nothing installed/configured
agent: 1
bios: ovmf
boot: order=scsi0;ide2;ide0
cores: 8
cpu: host
efidisk0: local-lvm:vm-1000-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:02:00,pcie=1
hostpci1: 0000:68:00,pcie=1,romfile=1080ti_patched.rom,x-vga=1
ide0: local:iso/virtio-win-0.1.229.iso,media=cdrom,size=522284K
ide2: local:iso/Windows10_22H2.iso,media=cdrom,size=4671872K
machine: pc-q35-6.2
memory: 16384
meta: creation-qemu=7.1.0,ctime=1680476494
name: w10
numa: 0
ostype: win10
scsi0: local-lvm:vm-1000-disk-1,iothread=1,size=100G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=53e117ca-52c6-462b-96a1-feb41d749b27
snaptime: 1680477590
sockets: 1
vga: none
vmgenid: bb5d3904-c42e-4206-b24e-b428df9ef584