GPU Passthrough issues

S

Simon9116

New Member
Aug 1, 2024
25
1
3
Hi everyone,
I am encountering issues with GPU passthrough in a Windows 11 VM.

My build:
- Ryzen 9 9950x (The integrated graphics is assigned to proxmox)
- Gigabyte x870 Gaming Wifi 6
- NVIDIA RTX 5070ti

UEFI settings:
- Resizable BAR: disabled
- Above 4G decoding: disabled
- CSM: disabled
- IOMMU: enabled

GRUB config:
GRUB_CMDLINE_LINUX_DEFAULT="quiet nomodeset initcall_blacklist=sysfb_init pcie_acs_override=downstream"
For anyone suggesting that I am missing amd_iommu=on iommu=pt please keep in mind that these parameters have been deprecated and are no longer valid (but invalid parameters are ignored so it doesn't matter).

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2c05,10de:22e9 disable_vga=1

/etc/modprobe.d/pve-blacklist.conf
blacklist nouveau
blacklist nvidia
blacklist nvidiafb
blacklist snd_hda_intel
Click to expand...

VM config:
args: -cpu host,-hypervisor,kvm=off, -smbios type=0,vendor="American Megatrends International",version=F5,date="03/12/2025"
bios: ovmf
boot: order=scsi0;net0
cores: 16
cpu: host,hidden=1
efidisk0: vms:vm-110-disk-0,efitype=4m,pre-enrolled-keys=1,size=1M
hostpci0: 0000:01:00,pcie=1,x-vga=1
machine: pc-q35-9.2+pve1
memory: 32768
meta: creation-qemu=9.2.0,ctime=1745758477
name: Win11-gaming
net0: virtio=BC:24:11:B9:D3:58,bridge=vmbr0,firewall=1
ostype: win11
scsi0: vms:vm-110-disk-1,iothread=1,size=150G
scsihw: virtio-scsi-single
smbios1: uuid=67dbbf21-678b-45dd-a5d4-94f1d3ac6487
sockets: 1
tpmstate0: vms:vm-110-disk-2,size=4M,version=v2.0
vga: virtio
vmgenid: c9eef372-e262-4aa4-83bf-b78268cda3a8
Click to expand...


Basically, I set up IOMMU correctly and I can pass the GPU to the VM. Then I configured the basic stuff (RDP, VirtIO drivers for the NIC, ...).

The issues arise when I try to download the nvidia drivers: at first the GPU seems to get recognized correctly (no code 43), but when I reboot it, it gets stuck on the loading screen (see image) and sometimes windows goes into recovery.
Further tests revealed that the issues start as soon as the drivers finish installing. I discovered this because I tried not to connect through RDP but with an HDMI cable to a physical monitor and there I could (sometimes) get video output until the drivers finished installing, then I could only see a black screen.

I already tried multiple "solutions" which did not work: changing the GRUB config, recreating the VM, removing the drivers through DDU and then reinstalling them even using NVCleanstall to check the MSI flag, but still no luck.

No errors in dmesg either, the only weird thing there (or at least I think it is) is that whenever I start the VM, it writes:
vfio-pci 0000:01:00.0: Enabling HDA controller
vfio-pci 0000:01:00.0: Enabling HDA controller
Click to expand...
I find it weird that it is enabling it twice.

Moreover, right after installing the driver (even before rebooting) windows in general becomes highly unstable, taking a lot of time to load apps and not registering clicks.
Plus, if I disconnect from RDP after the driver is installed and I try to reconnect, it prompts me to enter the credentials, but then fails to connect after a minute of loading.
This is why I think this isn't inherently a GPU passthrough problem, but more so a driver/compatibility one.

I still haven't tried passing through another card, however I have a 2060 super laying around from another computer that I could try installing on my current build to see if I can get that to work, but first I wanted to rule out every possible configuration problem.

Can someone help me?
 

Attachments

  • Immagine 2025-04-27 163449.png
    Immagine 2025-04-27 163449.png
    52.2 KB · Views: 7
Last edited:
hey,
For passingtrough my rtx3060, i only need theses 3 options:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Did you have tried with only theses options ?
 
After you installed the video-card driver, shut down the VM, edit the config:
Code: 
vga: none

Start the VM, done.
 
Last edited:
Pifouney said:
hey,
For passingtrough my rtx3060, i only need theses 3 options:
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"

Did you have tried with only theses options ?
Click to expand...
Hi, thank you for your response. I did try with only those options in the beginning, but it didn't work and also I need acs for isolation, otherwise other devices will be passed to the VM. The other options are there because I was reading some solutions on other posts on the forum but didn't work so I'll remove them.
 
Last edited:
  • Like
Reactions: Pifouney
How many physical video cards do you have in your HOST system?
If you have only 1-card and using the motherboard as the primary card, it will not work.
You need at least 2card:
-1card for your motherboard (primary)
-1card for passthrough.

The problem is, when you power on the HOST system it will be initialize the video-card (primary), and wont work as it should be. Thats why need the 2nd card to left unitialized, which will handle the VM when starts it.
 
emunt6 said:
How many physical video cards do you have in your HOST system?
If you have only 1-card and using the motherboard as the primary card, it will not work.
You need at least 2card:
-1card for your motherboard (primary)
-1card for passthrough.

The problem is, when you power on the HOST system it will be initialize the video-card (primary), and wont work as it should be. Thats why need the 2nd card to left unitialized, which will handle the VM when starts it.
Click to expand...
Yep, already considered it. In fact, I am using the CPU integrated graphics as the primary card.
 
i don't know if it will be of any help at all, but when trying to pass my Intel ARC A310 to my windows 10/11 VMs, with it being the primary GPU with no iGPU (using i7-7820X cpu and Gigabyte X299 ud4 mbo), i had to remove and rescan it to basically take it away from proxmox, since proxmox by default has control over it for terminal video output, etc as the main gpu. although it wouldn't really passthrough at all until i did this, so sounds like a totally different problem since you have gotten it to pass before the driver install, where as with mine it would error out since its the primary gpu.

here are the commands from my terminal, that i used in the process, its been some months since i did it so i don't remember perfectly but i went back in my history and grabbed the commands.

Code: 
echo 1 > /sys/bus/pci/devices/0000\:67\:00.0/remove # the hdmi audio controller
echo 1 > /sys/bus/pci/devices/0000\:68\:00.0/remove # the GPU
echo 1 > /sys/bus/pci/devices/0000\:68\:00.0/rescan # rescan GPU
echo 1 > /sys/bus/pci/rescan # rescan all
ls /sys/bus/pci/devices/0000\:68\:00.0

# replace 67-68 with device IIMOU group ID number

NOTE: if this is the problem, not all motherboards will allow it, some reboot instantly if the GPU is taken from the OS
 
Last edited:
Simon9116 said:
Thank you very much, but if I remember correctly I had already tried rescaning the GPU with no luck :(
Click to expand...
you're welcome, i hope you find the fix for this issue, it could just be that it is a blackwell card and there could be problems with support still since it is really new. but did you remove then rescan or just rescan?
 
  • Like
Reactions: Simon9116
zenowl77 said:
you're welcome, i hope you find the fix for this issue, it could just be that it is a blackwell card and there could be problems with support still since it is really new. but did you remove then rescan or just rescan?
Click to expand...
I think I removed it as well, but I'll try doing it again, it certainly won't hurt.
 
zenowl77 said:
i don't know if it will be of any help at all, but when trying to pass my Intel ARC A310 to my windows 10/11 VMs, with it being the primary GPU with no iGPU (using i7-7820X cpu and Gigabyte X299 ud4 mbo), i had to remove and rescan it to basically take it away from proxmox, since proxmox by default has control over it for terminal video output, etc as the main gpu. although it wouldn't really passthrough at all until i did this, so sounds like a totally different problem since you have gotten it to pass before the driver install, where as with mine it would error out since its the primary gpu.

here are the commands from my terminal, that i used in the process, its been some months since i did it so i don't remember perfectly but i went back in my history and grabbed the commands.

Code: 
echo 1 > /sys/bus/pci/devices/0000\:67\:00.0/remove # the hdmi audio controller
echo 1 > /sys/bus/pci/devices/0000\:68\:00.0/remove # the GPU
echo 1 > /sys/bus/pci/devices/0000\:68\:00.0/rescan # rescan GPU
echo 1 > /sys/bus/pci/rescan # rescan all
ls /sys/bus/pci/devices/0000\:68\:00.0

# replace 67-68 with device IIMOU group ID number

NOTE: if this is the problem, not all motherboards will allow it, some reboot instantly if the GPU is taken from the OS
Click to expand...
Okay, I tried Win10 and it has the same problem. I even tried to run all updates in win11 but still no luck.

I tried running the commands you gave me and it let me remove the gpu but not rescan it, I don't no why. It threw an error telling there was no such file or directory and I had to reboot the system to get the GPU to show up again. Maybe that can tell us something more about the issue.

Or maybe it's just like you said: the GPU is very new and maybe has some compatibility issues.
Thanks anyways.
 
Update: Still no fix

I tried updating the kernel to version 6.11.11-2 but that didn't help
I tried all driver versions that released since I first wrote the post but the problem persists, though honestly they were just hotfixes so I'll be waiting for a major driver release which will be dropping next week most likely.

I continued to search for a solution across forums but apparently I am the first person to encounter this problem (or at least the first to document it). I even tried asking deepseek and it cited my own post as a possible solution. I did found a post on another forum about problems with the new blackwell GPUs and passthrough (in general, not only on proxmox) but those issues were different from mine since they just had to update the drivers or disable ROM-BAR (which I tried to do btw).

If even the next driver doesn't work I will be plugging in my old GPU and try to passthrough that.
 
Last edited:
Actually, i think its the Nvidia cards and Proxmox not supporting properly the GPU PCIe pass through. They are not meant for pass through. I have a RTX4070.
Has not been able to pass through despite doing all the correct steps. The VM runs for max 15-30 mins and then not only does it crash, it also kills the Proxmox Host. 2nd reinstall already done and no luck still.

If you have any other way, do let me know.
 
mmilestone2 said:
Actually, i think its the Nvidia cards and Proxmox not supporting properly the GPU PCIe pass through. They are not meant for pass through. I have a RTX4070.
Has not been able to pass through despite doing all the correct steps. The VM runs for max 15-30 mins and then not only does it crash, it also kills the Proxmox Host. 2nd reinstall already done and no luck still.

If you have any other way, do let me know.
Click to expand...
Hi, thanks for your answer.

I actually don't think that's the case anymore since NVIDIA official started supporting GPU passthrough on consumer cards years ago and there are numerous cases of it working successfully on 40 series cards.

And I think my problem is different from yours since my VM freezes at boot only after the driver installation (beforehand passthrough works fine even if the OS is using windows basic display driver) and my proxmox host never had any problems nor crashed.

Anyways, I don't have a 40 series GPU so I don't really know how the passthrough situation is on those but from what I have read the 50 series is not yet ready for GPU passthrough due to drivers and reset shenanigans.
 
You need to pass through the entire NVIDIA card (which is typically a GPU and an HDMI audio)

0000:01:00.0 is a virtual function of the card, remove the last .0, make sure your Proxmox has it blacklisted and no nouveau or nVIDIA drivers there load.

All this romfile=vbios.bin,vendor-id=0x10de,x-vga=1 is completely unnecessary.

For ACS isolation issues, move the card to a different slot that doesn’t have isolation issues, any decent motherboard should have at least one x16 that’s not shared with other devices.
 
You must log in or register to reply here.
Top Bottom