PCI passthrough question (updated with more info)

[kbirger]

Hi all, I'm kind of baffled at what is going on for me. I had PCI passthrough working swimmingly, until one day it ceased. This lead me to once again spend several days re-reading all the different guides and people's experiences. There's some stuff that conflicts, because things are evolving, and a couple gaps.

I've installed the nvidia driver on my host with dkms, and installed the same driver without kernel modules in the guest. I'm not sure if this is actually correct, but it was previously working.

Currently, I'm in a situation where if I set up vfio, my nvidia driver fails to bind to the device during startup, because it is already bound to vfio-pci. This causes my system to not start up, and then I have to do a recovery boot and fiddle with modules.

The terminal is flooded with messages saying the following:
NVRM: GPU 0000:01:00.0 is already bound to vfio-pci

If I blacklist the nvidia module, then the system starts up, but nvidia-smi does not show any devices, and /dev/nvidia0 is not created, so no actual passthrough can happen.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. GA102 [GeForce RTX 3090] [19da:1613]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. GA102 High Definition Audio Controller [19da:1613]
Kernel driver in use: vfio-pci

This makes sense, I think.

If I search dmesg | grep -i vfio I still see the error, but only once, and it doesn't prevent the system from booting.

[   60.085356] nvidia-nvlink: Nvlink Core is being initialized, major device number 511
[   60.085374] NVRM: GPU 0000:01:00.0 is already bound to vfio-pci.
[   60.086640] NVRM: The NVIDIA probe routine was not called for 1 device(s).
[   60.086662] NVRM: This can occur when another driver was loaded and
               NVRM: obtained ownership of the NVIDIA device(s).
[   60.086675] NVRM: Try unloading the conflicting kernel module (and/or
               NVRM: reconfigure your kernel without the conflicting
               NVRM: driver(s)), then try loading the NVIDIA kernel module
               NVRM: again.
[   60.086695] NVRM: No NVIDIA devices probed.
[   60.107891] nvidia-nvlink: Unregistered Nvlink Core, major device number 511

This does NOT make sense. Not sure why it is loading it when the module is blacklisted.

If I instead remove the vfio modules from /etc/modules, then the system starts up, /dev/nvidia0 exists, nvidia-smi is happy, and the LXC that I have forwarded the device to is seeing the card. However, this also doesn't seem to do what I need, because it looks like other processes can't access the GPU anymore. This also makes sense.

So, how is this supposed to work? How do I allow the host to load the nvidia modules so that the device gets created, but not lock up the device so that passthrough can happen via vfio?

Additional info​

Host​

Proxmox: pve-manager/8.3.3/f157a38b211595d6 (running kernel: 6.8.12-7-pve)
Kernel: Linux kirstation3 6.8.12-7-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-7 (2025-01-17T08:18Z) x86_64 GNU/Linux

Boot: EFI systemd-boot

Kernel cmdline​

I've tried both with grub and with systemd-boot

/etc/kernel/cmdline
intel_iommu=on iommu=pt root=/dev/mapper/pve-root

/etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"

/etc/modules​

vfio
vfio_iommu_type1
vfio_pci

tried this in /etc/modules-load.d/ as well

lspci -knn​

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. GA102 [GeForce RTX 3090] [19da:1613]
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. GA102 High Definition Audio Controller [19da:1613]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

modprobe.d​

blacklist nouveau
options nouveau modeset=0


options md_mod start_ro=1
# This file contains a list of modules which are not supported by Proxmox VE

# nvidiafb see bugreport https://bugzilla.proxmox.com/show_bug.cgi?id=701
blacklist nvidiafb
#blacklist snd_hda_intel
blacklist nouveau\noptions nouveau modeset=0


#blacklist vfio_pci
#blacklist vfio_pci_core
#blacklist vfio_iommu_type1
#blacklist vfio

#blacklist nouveau
#blacklist nvidia
#blacklist nvidia-drm
#blacklist nvidia-modeset

options vfio-pci ids=10de:2204,10de:1aef
options vfio_iommu_type1 allow_unsafe_interrupts=1
#softdep nvidia pre: vfio-pci
#options nvidia-drm modeset=1

LXC (CT)​

Linux: Ubuntu 22.04.5 LTS

lxc conf​

arch: amd64
cores: 4
features: nesting=1
hostname: aihub
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BC:24:11:0F:96:0E,ip=dhcp,type=veth
onboot: 0
ostype: ubuntu
rootfs: local-lvm:vm-100-disk-1,size=100G
swap: 16384
template: 0
unprivileged: 0
lxc.mount.entry: /dev/vfio/devices/vfio0 dev/nvidia0 none bind,optional,create=file
lxc.cgroup2.devices.allow: c 237:* rwm

I've experimented with different lxc.mount.entry configurations, such as mapping the device paths 1:1, or not having a mount at all. 237 is the correct device ID for the vfio device, although I'm not sure if I'm meant to mount that or not. I currently don't have an nvidia device because I've blacklisted the drivers on the host.