Help with GPU passthrough...

[Volter]

Hi All,

I am having no luck getting GPU passthrough to work. I have followed the guidance and carefully looked through the threads. I have also watched a few YT videos. I can't figure out what I am doing wrong. I am on Proxmox 8.

My GPU (GTX1070) shows up in the VM device manager but with a code 43, disabling and reenabling it reoccurs. Windows 11/q35 vm/UEFI. I have installed the Nvidia drivers, which did install but control panel won't run.

My lspci -nnk shows me that the vfio-pci kernel driver is in use when the VM is on, it does not appear (I have no reference to drivers in use) when it is off.

# dmesg | grep -e DMAR -e IOMMU -e AMD-Vi - shows IMMOU enabled, Both GPU and Audio card are in their own IOMMU group.

I have added the GPU and Audio device ID to vfio.conf, I could see that my devices are listed with a subsystem ID related to my mainboard with an. I also added that to vfio.conf with no change. I have tried adding vga disabled here too.

I have blacklisted all the drivers, they do still appear under the devices as Kernel modules. The audio device did have snd_hda_intel as a driver in use but I also blacklisted that.

I have run out of things to try...

 

[LnxBil]

I have run out of things to try...

Moved the card into another slot? another mainboard?

In my experience: some constellation just don't work, whereas other just work out of the box. The gpu is recent enough so that it should work. I tried getting an older card to work with similar results than you, changed it to a 1080ti and it just worked out-of-the-box the first time. For passthrough of USB3 addon cards, I bought 3 and the last and most expensive one worked flawlessly, all others did not.

 

[Volter]

Moved the card into another slot? another mainboard?

In my experience: some constellation just don't work, whereas other just work out of the box. The gpu is recent enough so that it should work. I tried getting an older card to work with similar results than you, changed it to a 1080ti and it just worked out-of-the-box the first time. For passthrough of USB3 addon cards, I bought 3 and the last and most expensive one worked flawlessly, all others did not.

Yeah, the card has been in another windows PC and works fine. I haven't tried it in another slot, but I also have 1080 card I could try and squeeze in there to try that. I am using another slot to passthrough a sata controller, which works well. I could try swapping them, too many years putting GPUs in the top slot...

Another mainboard isn't an option (currently Asus x299), I would probably abandon GPU passthrough before I do that. I have another whole PC that I may use as a node one day, but I'll build experience with this one before I do that.

 

[Volter]

Just tried the 1080, same result. I can't try another slot as one isn't big enough for a two slot card and the other knocks out the sata connector that my VM SSD is on, I don't want to fully reconfigure it just yet.

One thing I have noticed if that when I update the initramfs I get the message saying "No /etc/kernel/pve-efiboot-uuids found, skipping ESP sync" from looking at threads I think this is because I am not running a ZFS system and it used grub as a bootloader? So this is not a bad message. However, could there be something here that I need to fix?

 

[leesteken]

If the GPU shows up inside the VM, then it's (most likely) not a IOMMU group or PCIe slot issue. It's a reset or driver issue. Try all combinations of Primary GPU, ROM-Bar, ROM-File, QEMU-machine version and NVidia driver versions. Check other threads for what kind of BIOS patching they needed for NVidia (I have no experience myself), and which QEMU-machine version that they sucessfully used with a certain NVidia driver version.

EDIT: If the GPU is used during the boot of the Proxmox host, then you need this work-around or drop the GPU from the PCIe bus and rescan (same thread).

 

[Volter]

Thanks for the post, I did play around with those switches but not very systematically so I can retry.

However when you say is it used during the boot of the host...yes. If I have a monitor plugged in I can see the splash screen and proxmox loads/ login option. The signal only stops when I start the VM it is mapped to.

 

[Volter]

If the GPU shows up inside the VM, then it's (most likely) not a IOMMU group or PCIe slot issue. It's a reset or driver issue. Try all combinations of Primary GPU, ROM-Bar, ROM-File, QEMU-machine version and NVidia driver versions. Check other threads for what kind of BIOS patching they needed for NVidia (I have no experience myself), and which QEMU-machine version that they sucessfully used with a certain NVidia driver version.

EDIT: If the GPU is used during the boot of the Proxmox host, then you need this work-around or drop the GPU from the PCIe bus and rescan (same thread).

Well... It turns out it works if I disable primary GPU..... I really thought I had tried that and then I remember, I did, but that was on Proxmox 7.3. I since updated to 8 to support this task but didn't check this again.

So I now have a Microsoft basic display adapter which is code 43, MS remote display adapter, which has always worked (I'm using RDP), and now the working 1070 too.

With primary GPU disabled, its essentially working with the standard settings as per the guidance file, I do have a monitor plugged into the card I it displays the windows lock screen when the VM is loaded, as well as the proxmox boot screen during host boot. It makes sense to me that it shouldn't be the primary GPU if I am accessing via RDP, but I did have a monitor plugged in the whole time and this never worked before.

I have run heaven benchmark to test and it is working, quite a lot of stutters, but working. I may still try the information on the other thread and see what's changes that makes, but at least I have a working basis. I didn't intend to play games on it so if this is as far as I get then I am happy. Nice bit of learning here for me too.

Thanks

 

[LnxBil]

Have you a BIOS setting for that? I read it in some BIOSes, that you can select the primary display (e.g. integrated one), such that your PCIe card will be secondary besides beeing the only gpu with an attached display? I know that I have this in my machine here in with the 1080ti is in.

 

[jasoncasey264]

Make sure the Primary GPU is OFF. (IDK what it does, but its nothing good.) And plug in a real monitor into your device hdmi port if you can. Reboot everything.

Sometimes the Graphics cards just really wanna be used for real.