PCIe Passthrough for HBA - blacklist driver question

[odonnell]

Hello, new member here, I finally made an account. I am also relatively new to Proxmox and still cutting my teeth; I have it running on a Dell R620 in a homelab environment. This has 8 local drive bays, through a H710 PERC card flashed with IT mode firmware. Works great, these drives comprise a pool I use for the VMs.

For my NAS, I have decided to go with TrueNAS in a virtual environment. I am setting up a Dell SC200 JBOD, and installed a LSI 9207-8e HBA card to interface with it. My plan is to pass through this entire card to the TrueNAS VM for simplicity.

Here is my concern: running lspci -v shows me that the mpt3sas module is being loaded for the H710 "HBA" and the 9207-8e card that I wish to pass through. Blacklisting the mpt3sas driver in a simple manner seems like it would also affect the H710, which I still need as mentioned earlier.

What is the best approach to blacklist mpt3sas, but only for this one PCIe device? I tried searching already but maybe I am not using the right terms, all of the solutions I am finding seem to be relatively complex, such as this one. Is there a simple option, possibly one where the module doesn't need to be blacklisted in the first place, or doesn't require quite so many changes to be made all over the place?

TIA!

 

[leesteken]

Instead of blacklisting, you can also bind the vfio-pci driver to the device instead of the actual driver. Just make sure to do it early in the boot process.
One way is to add vfio-pci.ids=xxxx:yyyy to the kernel command line. Another way is to add a file (named vfio.conf for example) to the /etc/modprobe.d/ directory that contains lines like:
options vfio-pci ids=xxxx:yyyy softdep mpt3sas pre: vfio-pci
Replace xxxx:yyyy with the two four-digit numbers you get for the device when using lspci -n.

 

[odonnell]

Thanks! I went with the option to add those 2 lines to vfio.conf, which I already made for my GPU passthrough.
Followed by update-initramfs -u. I did update with my device ID, 1000:0087.

However after a reboot, the lspci -v command reveals that the vfio_pci module was being loaded not just for this specified device, but also for the H710 PERC card, which was evident when none of my 'start at boot' VMs spun up after the VM pool was not found.

I also tried consolidating those 2 lines into 1 line:
options vfio-pci ids=xxxx:yyyy softdep mpt3sas pre: vfio_pci
And doing update-initramfs -u after.

Then the mpt3sas module is loaded once again, for both devices. What gives?

I should note that the system is already set up of GPU passthrough, Steps 1-4 of this guide are active.

 

[odonnell]

I should mention that I am an a neandarthal with only the most basic knowledge on any of this, learning as I go. If there is a better option such as passing through all 12 disks seperately to TrueNAS I would entertain that option as well.

 

[leesteken]

Thanks! I went with the option to add those 2 lines to vfio.conf, which I already made for my GPU passthrough.
Followed by update-initramfs -u. I did update with my device ID, 1000:0087.

However after a reboot, the lspci -v command reveals that the vfio_pci module was being loaded not just for this specified device, but also for the H710 PERC card, which was evident when none of my 'start at boot' VMs spun up after the VM pool was not found.

I did not expect this, sorry of the inconvenience. Can you show lspci -vn for both devices? This should only happen if both devices have the same device ID. Maybe there are more files in /etc/modprobe.d/ that have a line with vfio-pci ids=. Please try consolidating those lines and ids into one line in one file.

I also tried consolidating those 2 lines into 1 line:
options vfio-pci ids=xxxx:yyyy softdep mpt3sas pre: vfio_pci
And doing update-initramfs -u after.

Then the mpt3sas module is loaded once again, for both devices. What gives?

They need to be on separate lines and the second line might not be necessary. See the documentation for more information.

Giving 12 drives to a VM might be possible by adding lines to the VM configuration file like scsi13: /dev/disk/by-id/..... The drives will show up as QEMU-drives and SMART is not available at the VM level (only at the host), but it is much simpler.

 

[odonnell]

Thanks for the reply - you actually exposed something that I cannot believe I did not notice. The Device ID is the same for both cards! Which explains the last post I made. Weird, as one is a H710 mini mono (Dell proprietary form factor) and the other is a more generic LSI card.

Below are what lspci reports, 02:00 is the H710 (local 8 bays) and 41:00 is the HBA I want to pass through:

 

[leesteken]

I guess the Dell is a rebrand. This complicated matters for PCI passthrough: maybe bind both to vfio-pci and rebind one to mpt3sas later in the boot process? I do believe kernel developers are working on a way to be able to differentiate such devices, but I'm not sure when it will work in Proxmox.
Silly question: have you tried passing through the device while the mpt3sas driver is in use? Proxmox/VFIO will unbind the current driver and rebind to vfio-pci automatically. Often this just works. What was the problem or error message?
Do you need help passing the disks via /dev/disk/by-id/ to the VM? Maybe you could post your VMID.conf and the paths to the drives?

 

[odonnell]

Thank avw.

Thanks to USPS losing my package, the SFF-8088 cable is still not here, so I have yet to actually try passing it through with the JBOD connected. I am just basing this off of the other 'help' threads I found on the topic, where many users were having I/O issues due to the module being loaded by Proxmox and handed over to the VM at start. Weird issues. I will report back with what happened in my case as soon as the cable is here.

If it doesn't work out I will definitely have a go at passing through individual discs and update this thread.