[SOLVED] Issues with PCI Passthrough and NVIDIA card

[ducs4rs]

I picked up a Supermicro X9SRL-F installed Proxmox VE (pve-manager/8.3.1/fb48e850ef9dde27 (running kernel: 6.8.12-4-pve) and I am trying to get PCI passthrough to work on a NVIDIA Corporation GP106GL [Quadro P2200]. While the Video is getting set properly "Kernel driver in use: vfio-pci" the audio is not "Kernel driver in use: snd_hda_intel"

If I blacklist snd_hda_intel then I don't see "Kernel Driver in use: vfio-pci":

Capabilities: [100] Advanced Error Reporting
Kernel modules: snd_hda_intel

When I try to pass 2:00:1 through it gets an already in use message. If I leave it out, I get the dreaded MS 43 error on the video in Devices. Which may or may not be a result of the audio not being passed. Not sure where to go from here.

Here are the relevant configurations:

Kernel options: GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,57600n8 console=tty1 intel_iommu=on iommu=pt pci=assign-busses pci=nomsi pci=noaer pcie_aspm=off"

Blacklist:
blacklist nouveau
blacklist nvidia*
blacklist nvidiafb
blacklist radeon
blacklist snd_hda_intel

vfio.conf:
options vfio-pci ids=10de:1c31, 10de:10f1 disable_vga=1

lspci -nn:
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2200] [10de:1c31] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

Modules load:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

 

[leesteken]

I picked up a Supermicro X9SRL-F installed Proxmox VE (pve-manager/8.3.1/fb48e850ef9dde27 (running kernel: 6.8.12-4-pve) and I am trying to get PCI passthrough to work on a NVIDIA Corporation GP106GL [Quadro P2200]. While the Video is getting set properly "Kernel driver in use: vfio-pci" the audio is not "Kernel driver in use: snd_hda_intel"

If I blacklist snd_hda_intel then I don't see "Kernel Driver in use: vfio-pci":

blacklist does not early bind to vfio-pci. It just prevents the normal driver from binding automatically.

Capabilities: [100] Advanced Error Reporting
Kernel modules: snd_hda_intel

When I try to pass 2:00:1 through it gets an already in use message. If I leave it out, I get the dreaded MS 43 error on the video in Devices. Which may or may not be a result of the audio not being passed. Not sure where to go from here.

Please show the actual error message. Maybe you are confused about the All Functions option?

Here are the relevant configurations:

Kernel options: GRUB_CMDLINE_LINUX_DEFAULT="console=ttyS1,57600n8 console=tty1 intel_iommu=on iommu=pt pci=assign-busses pci=nomsi pci=noaer pcie_aspm=off"

Blacklist:
blacklist nouveau
blacklist nvidia*
blacklist nvidiafb
blacklist radeon
blacklist snd_hda_intel

Why blacklist radeon when you have an NVidia GPU? Why the additional pci= parameters? What are you trying to achieve with each of them?

vfio.conf:
options vfio-pci ids=10de:1c31, 10de:10f1 disable_vga=1

lspci -nn:
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106GL [Quadro P2200] [10de:1c31] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)

Don't put a space after the comma. Try options vfio-pci ids=10de:1c31,10de:10f1 disable_vga=1 instead. (Also try to use code tags for these things in the future.)

Modules load:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

vfio_virqfd no longer exists on recent Proxmox versions. Which old version are you using? Or an old guide on the internet? Please use the current admin manual and Wiki.

 

[ducs4rs]

Thanks for the pointers. The kernel parameters were from my notes when I first did pci passthrough on VE 5 or 6. I made all the changes you recommended and now the audio is correct,

02:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
...
 Kernel driver in use: vfio-pci

I am still getting error 43 from the windows nvidia driver. But the sound device is now functioning. Thanks for your help. Now to see if I can resolve the error 43..

 

[zenowl77]

while it is for vGPU setup not passthrough of the entire card, this guide here is usually pretty great to read over

PolloLoco / NVIDIA vGPU Guide

you can infer what is relevant to you if you do not want vGPU, you may also decide vGPU can be easier and better with a quadro card since you do not have to license them and can then use it on multiple VMs simultaneously

 

[ducs4rs]

while it is for vGPU setup not passthrough of the entire card, this guide here is usually pretty great to read over

PolloLoco / NVIDIA vGPU Guide

you can infer what is relevant to you if you do not want vGPU, you may also decide vGPU can be easier and better with a quadro card since you do not have to license them and can then use it on multiple VMs simultaneously

Thanks for the pointer. I am going to try it out.. I was able to pass the GPU and a VF nic through to Ubuntu. Windows 11 wasn't happy with either.

 

[alexvirt]

The same thing with an old Nvidia card (GTX 730): Audio passthrough works, but video GPU still produces "Error 43" at Windows 10 VM

 

[zenowl77]

Thanks for the pointer. I am going to try it out.. I was able to pass the GPU and a VF nic through to Ubuntu. Windows 11 wasn't happy with either.

you're welcome, windows 11 might give you the most issues out of all of them, but overall once you have IOMMU enabled and the right drivers it should just work (check your bios too of course to make sure you have everything enabled like vt-d, etc just to double check you have as much virtualization support as possible)

The same thing with an old Nvidia card (GTX 730): Audio passthrough works, but video GPU still produces "Error 43" at Windows 10 VM

you're definitely going to have a harder time with the older cards with anything that isn't simply full passthrough, but i have successfully passed a gt 610 so i know its possible.

make sure you have the right drivers,, i have had the most success using the KVM drivers forpassthrough, which aren't available for download publicly you have to find them online somewhere or make an account with NVIDIA as it explains in the vGPU post.

but they lack cuda, uvm and other host functions so that also means LXCs and other things wont work right with GPU you basically have to choose between consumer host gpu functions and vGPU vm functions.

 

[ducs4rs]

you're welcome, windows 11 might give you the most issues out of all of them, but overall once you have IOMMU enabled and the right drivers it should just work (check your bios too of course to make sure you have everything enabled like vt-d, etc just to double check you have as much virtualization support as possible)



you're definitely going to have a harder time with the older cards with anything that isn't simply full passthrough, but i have successfully passed a gt 610 so i know its possible.

make sure you have the right drivers,, i have had the most success using the KVM drivers forpassthrough, which aren't available for download publicly you have to find them online somewhere or make an account with NVIDIA as it explains in the vGPU post.

but they lack cuda, uvm and other host functions so that also means LXCs and other things wont work right with GPU you basically have to choose between consumer host gpu functions and vGPU vm functions.

I did the vcpu and it works like a charm.

 

[zenowl77]

good to hear, glad you got it working