[SOLVED] RTX 4090 Passthrough : AMD, Proxmox and sadness

[comma]

Hi everyone,

I'm losing my sanity a bit on this topic and I would like your previous help, let's begin!

Spoiler: lspci -vv (BEFORE SETUP)

# lspci -vv and info on the device I want to passthrough
0f:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1) (prog-if 00 [VGA controller])
Kernel modules: nvidiafb, nouveau

0f:00.1 Audio device: NVIDIA Corporation AD102 High Definition Audio Controller (rev a1)
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

Spoiler: lspci -nnv | grep VGA

lspci -nnv | grep VGA
================
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1) (prog-if 00 [VGA controller])
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090] [10de:2684] (rev a1) (prog-if 00 [VGA controller])

Spoiler: dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi
================================================
[ 0.276619] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[ 0.714320] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.718366] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.718368] AMD-Vi: Extended features (0x58f77ef22294a5a, 0x0): PPR NX GT IA PC GA_vAPIC
[ 0.718371] AMD-Vi: Interrupt remapping enabled
[ 0.718591] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

Spoiler: dmesg | grep 'remapping'

dmesg | grep 'remapping'
====================
[ 0.442503] x2apic: IRQ remapping doesn't support X2APIC mode
[ 0.718371] AMD-Vi: Interrupt remapping enabled

Spoiler: cat /etc/modules

cat /etc/modules
==============
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
# Parameters can be specified after the module name.
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Spoiler: modprobe files

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia*" >> /etc/modprobe.d/blacklist.conf
echo options vfio-pci ids=10de:2684,10de:22ba disable_vga=1 > /etc/modprobe.d/vfio.conf

Spoiler: Inside Ubuntu VM

01:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 165b
Physical Slot: 0
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

I passed the pcie as per the doc and installed Nvidia drivers from Nvidia, no nvidia-smi available but the driver seems there, also the gpu name doesn't appear on lspci but only the code

​

​

FIXED​

It was secure boot on the ubuntu VM ​

Just disable it and it works ​