[SOLVED] PCI Passthrough Blacklisting

[tipex]

Hi all
I've had a search around and cant seem to find the answer. I basically want to know if I have correctly done blacklisting and if its working. I have an LSI sas card in my server which I am passing through to a VM so the VM can use the hard drive that is plugged into the SAS card. This seems to be working.
The bit I am concerned with is blacklisting. I want to ensure that this card is only ever used by that one VM and not by any other VMs or by proxmox itself.
In the following guide there are two methods for this (https://pve.proxmox.com/wiki/PCI(e)_Passthrough):
1 - Add the device ID to a new conf file at /etc/modprobe.d/somefile.conf
2 - Adding the drivername to the existing /etc/modprobe.d/pve-blacklist.conf
For option 2, I am not sure where to get the drivername so instead I did option 1. Using the follwoing command I find the device ID (in my case b6:00):

lspci -v

Then using the following command I find some other characters that are needed in the conf file:

lspci -n -s b6:00

I then added the following into the new conf file:

options vfio-pci ids=1000:0072

After this I run
update-initramfs -u -k all
and reboot
So far so good but I would like to know if the blacklisting is working.
Is there an obvious way to test if blacklisting is working?

 

[tipex]

I think I may of found the answer. If anyone can confirm the below is true that would be great - if not I'm fairly confident its working.
Running the following command gives details about devices:

lspci -v

With the VM (that has the card passed through to it) switched off I get the following:

I

If I switch the VM on and run the command again I get:

From what I have read online this means that the card is only passed through when the VM is switched on. The proof is that the kernal driver is in use by vfio-pci
Based on this I think I am happy blacklisting is working.
Now to move onto my next Proxmox task whcih is clustering. I love this platform!

 

[dcsapak]

no it seems the driver blacklisting does not really work, since the kernel still uses the 'mpt3sas' driver before you start the vm (this is the driver name you looked for btw)
if it works like this, you can also leave it like that, since we try to remove the device from the original driver and assign it to the 'vfio-pci' automatically
since that does not work for every device/driver, the notes in the docs exist

 

[Dunuin]

if it works like this, you can also leave it like that, since we try to remove the device from the original driver and assign it to the 'vfio-pci' automatically
since that does not work for every device/driver, the notes in the docs exist

Also wanted to blacklist the Device/vendorID or at least backlist the mpt3sas driver. Here ZFS is used on the HBAs and I don't want PVE to import the ZFS pools stored on the HBA cards disks. I think I heard there is a way to tell ZFS to not import specific pools on boot, but can not find that source again.

Right now I added this to my fvio.conf looks like this to blacklist my two GPU functions: echo "options vfio-pci ids=10de:128b,10de:0e0f disable_vga=1" > /etc/modprobe.d/vfio.conf. To also add the two HBAs, I would just need to add more device/vendorIDs to it like this and then yupdate the initramfs/bootloader?:
options vfio-pci ids=10de:128b,10de:0e0f,xxxx:xxxx,yyyy:yyyy disable_vga=1

And that would also prevent initramfs from initializing these HBAs?

 

[tipex]

Damn - just when I thought I had solved it.
@Dunuin your line is very similar to mine only with different device IDs obviously and you also have the disable_bag=1 bit which I dont as I'm not passing any GPUs so I dont think I need that.
I'm basically trying to pass my LSI SAS card to a TrueNAS VM (dont worry its a home server not production). I'd obviosuly like blacklisting to be fully working to avoid any possible failures with ZFS etc in TrueNAS.
I wonder why its not working as the line is quite simple. Its not like there are loads of options.

 

[dcsapak]

my guess is that the mpt3sas module is loaded before the vfio-pci one and thus takes precedence, you could check if manually loading vfio-pci before the mpt3sas works by adding a 'softdep' line in the modprobe config
see 'man modprobe.d' for details on how the line must look like

 

[Dunuin]

You didn't forget the update-initramfs -u -k all and the reboot afterwards, so that these config changes actually take into action?

 

[tipex]

Yep I'm using update-initramfs -u -k all each time and also rebooting proxmox.

I have tried many combinations of adding the device ID to:
/etc/default/grub
/etc/kernel/cmdline
/etc/modules
/etc/modprobe.d/pve-blacklist.conf
/etc/modprobe.d/vfio.conf

In the format of:
options vfio-pci ids=1000:0072
vfio-pci ids=1000:0072
vfio-pci.ids=1000:0072
Also tried with underscore rather than dash as I saw underscore being used on some search results.

For the softdep in /etc/modprobe.d/pve-blacklist.conf I'm not sure what the name should be before pre:....

softdep some_name pre: vfio-pci

I'd guess at mpt3sas:

 

[dcsapak]

yes that's the driver name you're looking for

 

[tipex]

Adding softdep mpt3sas pre: vfio-pci to /etc/modprobe.d/pve-blacklist.conf looks to have worked. With the VM switched off I see the following. I just need to double check what other lingering lines I have inplace across the many config files I have been messing with. Let me tidy things up and report back:

 

[tipex]

Ok so the changes I have made are:

/etc/modprobe.d/vifo.conf:
options vfio-pci ids=1000:0072

/etc/modprobe.d/pve-blacklist.conf:
softdep mpt3sas pre: vfio-pci

From what I've read online you either use one option or the other. So it seems for my machine the vifo conf file did not do anything but adding the softdep line did do something. I can therefore probably remove vifo.conf
I will try that now and see what happens.

 

[tipex]

Interesting. It seems you need both files in place for blacklisting to work. This (https://pve.proxmox.com/wiki/PCI(e)_Passthrough) indicates one or the other but not both.

Any way I think its working now.
@dcsapak Should we mark this as solved?

 

[dcsapak]

@dcsapak Should we mark this as solved?

yes you can do that yourself by editing the thread (there should be a 'solved' prefix for the title to choose from)

 

[Volter]

Just picking up on this thread, I am having no luck getting GPU passthrough to work. I have followed the guidance and carefully looked through the threads. My GPU shows up in the VM device manager but with a code 43, disabling and reenabling it reoccurs. Windows 11 - q35 vm

My lspci -nnk shows me that the vfio-pci kernal driver is in use when the VM is on, it does not appear (i have no reference to drivers in use) when it is off.

I have added the GPU and Audio device ID to vfio.conf, I could see that my devices are listed with a subsystem related to my mainboard with an id. I added that with no change.

I have blacklisted all the drivers, they do still appear under the devices as Kernel modules. The audio device did have snd_hda_intel as a driver in use but I also blacklisted that.

Both GPU and Audio card are in their own IOMMU group.

I have run out of things to try...

 

[dcsapak]

Just picking up on this thread, I am having no luck getting GPU passthrough to work. I have followed the guidance and carefully looked through the threads. My GPU shows up in the VM device manager but with a code 43, disabling and reenabling it reoccurs. Windows 11 - q35 vm

My lspci -nnk shows me that the vfio-pci kernal driver is in use when the VM is on, it does not appear (i have no reference to drivers in use) when it is off.

I have added the GPU and Audio device ID to vfio.conf, I could see that my devices are listed with a subsystem related to my mainboard with an id. I added that with no change.

I have blacklisted all the drivers, they do still appear under the devices as Kernel modules. The audio device did have snd_hda_intel as a driver in use but I also blacklisted that.

Both GPU and Audio card are in their own IOMMU group.

I have run out of things to try...

did you try dumping the GPU rom and providing that via the config? error 43 is a very generic error and can have many causes: https://pve.proxmox.com/wiki/PCI_Passthrough#Error_43