Code 43 - passthrough of NVIDIA RTX 2080 in windows 10/11 VM

[asaheed]

Hi Guys,

I just installed Proxmox for the first time and aimed to virtualize my workstation / gaming windows environment in the first place. I guess this should work fine as NVIDIA opened up the passthrough of consumer cards and hoped it wouldn't be as painful as some years ago when I did the same with Unraid. Well, as you already read in the headline it didn't went to plan at all.

But first here is the hardware/system:
- Mainboard ASRock Fatal1ty X399
- AMD Threadripper 2950X
- 64 GB DDR 4 RAM
- Multiple NVMEs, SSDs and HDDs (Proxmox is installed on one of the nvme drives)
- NVIDIA RTX 2080
- Proxmox 7.0-11, all updates applied

What I've done so far:
1. Enabled IOMMU in the BIOS

2. Updated Grub:
- added the following line to nano /etc/default/grub :
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
later I also added the following because someone mentioned it to me it might help (splits the pci groups up even more?):
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction video=efifb:eek:ff"
- performed a update-grub

... rebooted

3. Validated IOMMU is active:
dmesg | grep -e DMAR -e IOMMU
Result:
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA [ 0.818008] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported [ 0.818051] pci 0000:40:00.2: AMD-Vi: IOMMU performance counters supported [ 0.849183] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.849193] pci 0000:40:00.2: AMD-Vi: Found IOMMU cap 0x40 [ 0.850793] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank). [ 0.850809] perf/amd_iommu: Detected AMD IOMMU #1 (2 banks, 4 counters/bank).

4. added required modules in /etc/modules:
vfio vfio_iommu_type1 vfio_pci vfio_virqfd

5. As this is the online graphics card in this system, I blacklisted the drivers in /etc/modprobe.d/pve-blacklist.conf so Proxmox won't use the card:
blacklist nvidiafb blacklist nvidia blacklist radeon blacklist nouveau

6. I check the pci address of the card with lspci -v | grep NVIDIA:
42:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] 42:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) Subsystem: NVIDIA Corporation TU104 HD Audio Controller 42:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI]) Subsystem: NVIDIA Corporation TU104 USB 3.1 Host Controller 42:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) Subsystem: NVIDIA Corporation TU104 USB Type-C UCSI Controller

7.I added the following line to /etc/modprobe.d/vfio.conf to be really sure the card isn't used by the host:
options vfio-pci ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9 disable_vga=1

8. I rebooted once more.

9. I crated a windows 10 VM with the following settings:
agent: 1 bios: ovmf boot: order=ide0;ide2;net0 cores: 8 efidisk0: local-lvm:vm-101-disk-1,size=4M hostpci0: 0000:42:00,pcie=1,x-vga=1 ide0: local-lvm:vm-101-disk-0,size=100G ide2: local:iso/virtio-win.iso,media=cdrom,size=543272K machine: pc-q35-6.0 memory: 16384 name: windows10 net0: e1000=66:F9:37:39:69:1E,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsihw: virtio-scsi-pci smbios1: uuid=5dc80c34-957d-4660-8d7e-70bc877ef189 sockets: 1 vga: none vmgenid: f4b10702-295f-41fa-af17-b3b3dbaf88a5

10. Windows found, correctly identified the card as a RTX 3080 and installed a driver from one year ago. (No Code 43)

11. I installed the current driver from the NVIDIA Website

12. The Card is now flagged with Code 43

I also tried this with Windows 11 with the same result.

After googling and experimenting for hours I need some help

Anyone got an idea what I missed/did wrong or what could cause the issue?

Thanks in advance,

asah

 

[asaheed]

Just to let you know:

I was confused because of the Intel USB Controller issues and it seems to be a problem of the windows 10 VM only, even when the RTX2080 is removed. My windows 11 VM doesn't have such issues with these usb controllers. not with and not without the NVIDIA card passed through.

 

[asaheed]

one more wired thing I recognized while trying to solve the issue:

I got a tip to try to "hide" that it's a virtual machine by setting the cpu type to "host" for the vm. after this had no effect (windows still showed kvm processors in device and task manager, I tried the following line in the vm conf file:
cpu=host,hidden=1,flags=+pcid
This also had no effect.

 

[asaheed]

I just read a tutorial which mentioned lspci -v will show me if the gfx card has now vfio drivers so I executed the command and this is the result:
42:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] Flags: bus master, fast devsel, latency 0, IRQ 11, IOMMU group 43 Memory at 9d000000 (32-bit, non-prefetchable) [size=16M] Memory at 80000000 (64-bit, prefetchable) [size=256M] Memory at 90000000 (64-bit, prefetchable) [size=32M] I/O ports at 3000 [size=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?> Capabilities: [900] Secondary PCI Express Capabilities: [bb0] Physical Resizable BAR Kernel driver in use: vfio-pci Kernel modules: nvidiafb, nouveau 42:00.1Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) Subsystem: NVIDIA Corporation TU104 HD Audio Controller Flags: bus master, fast devsel, latency 0, IRQ 10, IOMMU group 44 Memory at 9e080000 (32-bit, non-prefetchable) [size=16K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [100] Advanced Error Reporting Kernel driver in use: vfio-pci Kernel modules: snd_hda_intel 42:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI]) Subsystem: NVIDIA Corporation TU104 USB 3.1 Host Controller Flags: fast devsel, IRQ 161, IOMMU group 45 Memory at 92000000 (64-bit, prefetchable) [size=256K] Memory at 92040000 (64-bit, prefetchable) [size=64K] Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Power Management version 3 Capabilities: [100] Advanced Error Reporting Kernel driver in use: xhci_hcd Kernel modules: xhci_pci 42:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) Subsystem: NVIDIA Corporation TU104 USB Type-C UCSI Controller Flags: bus master, fast devsel, latency 0, IRQ 118, IOMMU group 46 Memory at 9e084000 (32-bit, non-prefetchable) [size=4K] Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Endpoint, MSI 00 Capabilities: [b4] Power Management version 3 Capabilities: [100] Advanced Error Reporting Kernel driver in use: nvidia-gpu Kernel modules: i2c_nvidia_gpu

I am not sure if the Kernel Drivers for the USB Controller and the Serial bus controller are how they should be, as most of the stuff I find online talks about the 10-series which did not have these two. Anyone got any experience if anything is correct here?