[SOLVED] Nvidia RTX 3060 Passthrough woes

[MrGeezer]

Hi,

This is solved. My system spec for anyone who stumbles on this post:

Intel NUC 11 "Beast Canyon" i9
64 Gb RAM
NVIDIA GTX 3060


I have an intel NUC 11 with core i9, 64Gb RAM, GeForce RTX 3060 running Proxmox 7.2-3. I want to pass the card through to a windows 10 pro guest VM following the various guides posted on the web.

Ive set /etc/kernel/cmdline to root=ZFS=rpool/ROOT/pve-1 boot=zfs intel_iommu=on iommu=pt video=efifb:off and /etc/modules to contain

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

I have checked the card is in an IOMMU group by itself (it is) and blacklisted nvidia in /etc/modprobe.d/blacklist.conf. The contents of my /etc/modprobe.d/vfio.conf is

options vfio-pci ids=10de:2503,10de:228e disable_vga=1

When I run dmesg | grep -e DMAR -e IOMMU I can see a line that says 0.089960] DMAR: IOMMU enabled

I have created a Windows10 VM. OVMF Bios, q35 machine with processor set to "HOST". I then select the pci device 0000:01:00 as a PCI device on the VM and try to boot.

This results in the VM hanging.

I've had this working (and opened a thread when it stopped) and after numerous reinstalls of Proxmox, creation of new VM's etc and a brief period of sucess when the VM ran fine for about 6 hours yesterday, it's not back to refusing to complete booting. If I attach a VGA card and run the console I can see it getting stuck as it boots - I get the proxmox logo and then the spinning windows logo appears, spins once or twice and freezes.

I know this is unsupported and I'm about to give up and go back to bare metal but by any chance can anyone point to where I could go to see any further clues what's going wrong? Is there a log file somewhere in proxmox I can read for some information as to what went wrong?

Thanks,

 

[MrGeezer]

I managed to find someone suggesting dmesg. This showed errors to do with "bar0 can't reserve" which maybe meant that the PVE host had been using the card?

After more reading I decided to install proxmox v6 from an ISO to see if that worked. Terrible plan as I was unable to get the bridge networking to function at all, so never got as far as adding the GPU.

I reinstalled PVE 7.2-3 from the ISO and this time I remembered that there was one tiny thing that had changed since the first time I got this working (had it working for around 2 days non-stop initially) - the first time I did this I used ext4 so proxmox configured GRUB. In later tries I switched to ZFS so I would get systemd boot as it matched a tutorial from reddit I was following.

Sooo, this time I installed on ext4 again and setup the intel_iommu thing in the grub file instead of the systemd file (following instructions from proxmox documenation so I could deviate from the tutorial which used systemd)

Result - VM boots fine. Installed NVIDIA driver 516.40.

I suspect this will stop working at some point - the longest it's worked so far is about two days. I've managed to take a snapshot of the working VM. I would back it up, but one of the things I have on the list of possible causes for it breaking is when I add a Proxmox Backup Server and/or back the machine up.

I have also disabled windows update entirely in the VM as that's also on the list of possible causes.

 

[leesteken]

BAR can't reserve memory errors are indeed typically because the GPU is used during boot. With 5.15 kernels, this typically happens when booting in UEFI mode where BOOTFB does not release its claim in /proc/iomem, even when using video=simplefb:off. The most elegant work-around I know is described here.
I also read on this forum that there are issues with recent NVidia drivers for Windows, but I have no experience with them.

 

[MrGeezer]

Thank you. I've woken this morning to find guest crashed and dmesg revealing BAR1 can't reserve. I will read the link this evening and see if I can fix it thank you very much for youe help

 

[MrGeezer]

Possible progress here - after readying the link provided by @leesteken I added initcall_blacklist=sysfb_init to /etc/default/grub.

Partial sucess - the VM now boots without hanging and I am able to install the last-but-one NVIDIA driver.

However - after 'a while' (Am at work so can't keep constant watch but it's around 1 hour) the VM powers itself off! It can be powered back on and runs again without hanging. But will soon power itself off.

When I examine the dmesg command on the host I no longer see BAR CAn't reserve errors but I do get the following which seems like it might be relevant/concerning?

14528.845063] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00
[14528.845991] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x2922
[14528.865089] vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x25@0x160
[14532.190623] vfio-pci 0000:01:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x2922
[14537.194440] x86/split lock detection: #AC: kvm/34762 took a split_lock trap at address: 0xfffff8045000df4c
[14542.383502] kvm [34675]: vcpu7, guest rIP: 0xfffff804501f76f2 vmx_set_msr: BTF|LBR in IA32_DEBUGCTLMSR 0x1, nop

** EDIT** Aww I'm not so sure now. I've had the WIN10 VM running for about 3 hours now. Maybe it's fine and it rebooted for some other reason I just don't know. Is there any way of stress testing the VM? I guess just load my steam game and play for a few hours.

 

[MrGeezer]

God I am a huge idiot. I got so fed up with rebuilding my VM I couldn't be bothered to activate my windows. The ever helpful microsoft are shutting my software down as they think it's expired/pirated!

The fix provided by @leesteken above works flawlessly.

In an effort to go some way to repaying the karma, I will update my original post with solved, and add as much detail about by system as possible in case anyone finds this thread via google one day. The solution to my problem was to add initcall_blacklist=sysfb_init to /etc/default/grub.

Thank you