[SOLVED] GPU Passthrough Nvidia 4090 Issues

[Riffy]

Good afternoon everyone. So for the last week I have been trying to pass the gpu I have to a windows 10 vm I setup. The gpu is an msi 4090. So far I have had nothing but failures and problems with it. First off I wanted to ask if loading the 6.1 kernel update maybe a good idea or not? As for the rest here are some images of how I loaded the vm https://www.dropbox.com/sh/r0cnn2mmj2dgrs0/AAABINE05Y_O40UuScpGcMMta?dl=0 and I added the pci device for the video card and ticked three and left the primary gpu one unchecked. I would then install windows like normal boot into it and download the nvidia drivers. During or sometimes just at the end of the driver install the system freezes. I end up needing to stop it to get it to load again and it boots into recovery mode then a reboot again to get it to boot. Now if the gpu is the only one or I tag it as primary gpu and disable the default one it never seems to boot. I am having to guess it's not booting since vnc, parsec, or rdp can't connect to it. If I disable the video card and re enable the default one I can login fine again.

I was told to add this to the end of the conf file for the vm "args: -fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536" which I added to the bottom of the file but still didn't make any progress. I only partly understand it but I was told it's needed for video cards with the resizable bar stuff on them.

So at this point does anyone have any advice or can point out my mistake in getting this working?

Oh also since this will be pointed out, I did edit the grub with amd_iommu=on, added the 4 VFIO modules to the modules file, and blacklisted amd, nvidia, and intel video drivers so proxmox won't use the gpu.

Also sorry about the wall of text. Trying to remember everything I have done/tried so far.

 

[leesteken]

amd_iommu=on does nothing as it is on by default on AMD. No need to blacklist AMD, NVidia and Intel drivers, since you have not installed NVidia drivers on the Proxmxo host (I hope, otherwise remove them or reinstall Proxmox) and the others don't apply.
Sometimes it helps to disable or enabled above 4G Decoding and Resizable BAR is not yet supported (I think), so please disable that in the BIOS.
It's not clear to me which options you "ticked" but the defaults usually work and for NVidia you do want to enable Primary GPU (but not for other brands).
It sounds like you have only one GPU in the system and you most likely need this work-around: so add initcall_blacklist=sysfb_init and check with cat /proc/cmdline whether it is applied after a reboot.
Lots of threads on this forum about PCI(e) passthrough, including modern NVidia GPUs and (old and no longer functional work-arounds for single GPU systems).

 

[Riffy]

amd_iommu=on does nothing as it is on by default on AMD. No need to blacklist AMD, NVidia and Intel drivers, since you have not installed NVidia drivers on the Proxmxo host (I hope, otherwise remove them or reinstall Proxmox) and the others don't apply.
Sometimes it helps to disable or enabled above 4G Decoding and Resizable BAR is not yet supported (I think), so please disable that in the BIOS.
It's not clear to me which options you "ticked" but the defaults usually work and for NVidia you do want to enable Primary GPU (but not for other brands).
It sounds like you have only one GPU in the system and you most likely need this work-around: so add initcall_blacklist=sysfb_init and check with cat /proc/cmdline whether it is applied after a reboot.
Lots of threads on this forum about PCI(e) passthrough, including modern NVidia GPUs and (old and no longer functional work-arounds for single GPU systems).

Thank you for the reply and a lot of what I am doing is based off old reddit posts and some new ones I made. People swear to me you needed the iommu on even after reading in the docs you don't need it for amd. I never did load any drivers inside of proxmox itself so I again just followed what I was told needed to be done by blacklisting. As for what I ticked, sorry it was in the images. I picked all funtions, rombar and pcie. Do you pick primary GPU before or after you have installed the drivers and disabled the default video device in the hardware tab in proxmox for the vm?

Yes I am working on only one gpu for the moment the passive card turns up this week. So I thought just popping the card in with a dummy jack in the display port would work. I only ended up posting in here cause I was getting burnt out trying to figure it out and getting a lot of mixed information from people or very out of date posts. But I will reread the proxmox docs again and try again.

 

[leesteken]

I picked all funtions, rombar and pcie. Do you pick primary GPU before or after you have installed the drivers and disabled the default video device in the hardware tab in proxmox for the vm?

All Functions is good (and no need to add the other functions as separate passthrough devices). PCI Express is also good (as long as you use the Q35 machine type). ROM Bar is most likely good. No need to set Display to None, Primary GPU does that for you. Just use it from the beginning.
I cannot comment on the NVidia drivers, as I have no experience with them (since they broke passthrough on purpose in the past). Lot's of forum post here though.

Yes I am working on only one gpu for the moment the passive card turns up this week. So I thought just popping the card in with a dummy jack in the display port would work. I only ended up posting in here cause I was getting burnt out trying to figure it out and getting a lot of mixed information from people or very out of date posts. But I will reread the proxmox docs again and try again.

Just follow the manual and bind all functions of the devices to vfio-pci (like it sounds like you already did). But you really need the initcall_blacklist=sysfb_init with one GPU (which is not properly released since kernel version 5.15). Make sure a physical display is connected to see the output (and enable output).

 

[Riffy]

All Functions is good (and no need to add the other functions as separate passthrough devices). PCI Express is also good (as long as you use the Q35 machine type). ROM Bar is most likely good. No need to set Display to None, Primary GPU does that for you. Just use it from the beginning.
I cannot comment on the NVidia drivers, as I have no experience with them (since they broke passthrough on purpose in the past). Lot's of forum post here though.

Just follow the manual and bind all functions of the devices to vfio-pci (like it sounds like you already did). But you really need the initcall_blacklist=sysfb_init with one GPU (which is not properly released since kernel version 5.15). Make sure a physical display is connected to see the output (and enable output).

Can I buy you a beer, I got it working and still don't believe it. It was so simple this time and everything just worked, I am now waiting for it to burst into flames or something as punishment from the gods or something.