vfio used for a device it shouldn`t be used for

[bolzerrr]

Hello,

i have following configuration:
cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs mitigations=off intel_iommu=on iommu=pt i915.enable_guc=3 i915.max_vfs=3

cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=544d:6178

And that device get the correct module:
lspci -nnk | grep -A 3 02:00.0

02:00.0 Multimedia controller [0480]: TBS Technologies DVB Tuner PCIe Card [544d:6178]
Subsystem: TBS Technologies (wrong ID) DVB Tuner PCIe Card [6909:0001]
Kernel driver in use: vfio-pci


However for some reason vfio is used for my nvme with a totally different id:

lspci -nnk | grep -A 3 01:00.0

01:00.0 Non-Volatile memory controller [0108]: Micron Technology Inc 2300 NVMe SSD [Santana] [1344:5405]
Subsystem: Micron Technology Inc 2300 NVMe SSD [Santana] [1344:0100]
Kernel driver in use: vfio-pci
Kernel modules: nvme



Which should not be the case. How can this happen and how to prevent?

 

[gfngfn256]

cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=544d:6178

I think (never done it myself) that you missed a dot in that line, so it should really be:

options vfio-pci.ids=544d:6178

I believe this has caused it to only use vfio-pci without passing the id.

 

[bolzerrr]

As kernel command line a dot is used. I added the subdevie id to harden it but still the nvme gets vfio-pci:

cat /etc/kernel/cmdline
root=ZFS=rpool/ROOT/pve-1 boot=zfs mitigations=off intel_iommu=on iommu=pt vfio-pci.ids=544d:6178:6909:0001 i915.enable_guc=3 i915.max_vfs=3

 

[leesteken]

Do you (automatically) start a VM that has the NVMe passed through? Then Proxmox bind vfio-pci to the device (until a reboot or manually rebinding the right driver).
Do you have other .conf files in /etc/modprobe.d/ maybe? Any other scripts or cron jobs that might change the driver?
EDIT: Did you install and use driverctl in the past, maybe?

 

[gfngfn256]

As kernel command line

Have you tried your original configuration (first post) but with the dot I mentioned?

 

[bolzerrr]

Have you tried your original configuration (first post) but with the dot I mentioned?

Yes thats wrong syntax:
update-initramfs -u -k all
update-initramfs: Generating /boot/initrd.img-6.8.8-2-pve
libkmod: ERROR ../libkmod/libkmod-config.c:712 kmod_config_parse: /etc/modprobe.d/vfio.conf line 1: ignoring bad line starting with 'options'

Do you (automatically) start a VM that has the NVMe passed through? Then Proxmox bind vfio-pci to the device (until a reboot or manually rebinding the right driver).
Do you have other .conf files in /etc/modprobe.d/ maybe? Any other scripts that might change the driver?

I don`t want to passthrough the nvme at all. But somehow as soon as i use vfio-pci the nvme get the module

 

[bolzerrr]

I figured that the behaviour changes once i disable sr-iov for my GPU. Strange enough that the second nvme is not affected and working also with sr-iov in place.

04:00.0 Non-Volatile memory controller [0108]: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 [2646:5013] (rev 01)
        Subsystem: Kingston Technology Company, Inc. KC3000/FURY Renegade NVMe SSD E18 [2646:5013]
        Kernel driver in use: nvme
        Kernel modules: nvme

There is also a module param but it doesnt have any impact in my situation:

cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=544d:6178:6909:0001
options vfio-pci enable_sriov=1

I made a udev rule that does fix the issue:

ACTION=="add|change", SUBSYSTEM=="pci", ATTRS{vendor}=="0x1344", ATTRS{device}=="0x5405", RUN+="/bin/sh -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind; echo 0000:01:00.0 > /sys/bus/pci/drivers/nvme/bind'"

But seems to be too early or too late during boot, there i create a systemd:

[Unit]
Description=Rebind NVMe Device to nvme Driver
DefaultDependencies=no
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/unbind; echo 0000:01:00.0 > /sys/bus/pci/drivers/nvme/bind;swapon /dev/disk/by-id/nvme-eui.000000000000000100a075212fb4ac3c-part5'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

I don't like it too much but it is doing the job. It would be great if vfio-pci would stick to the configuration and only pick the device configured.