Blacklisting a specific PCIe device (HBA)

[alex3025]

Hello, I have a DELL R740xd server with 2 HBA330 controllers connected to the backplane. This server supports the "split-backplane" feature which allows to assign the disks to a certain controller.

I wanted to use the first controller for PVE boot and VM/CT storage and the second controller as storage for a single VM. To do that, I've assigned the raw device (HBA) to my VM and it's working but I'm worried about what could happen on a host reboot or events like that.

Now, I read that when doing things like this (or passthrough in general) you should blacklist the device driver from the host but in this case I can't because then both controllers will be unavailable to the host (so Proxmox cannot boot).

Any suggestions?

 

[daanw]

You can use udev rules to bind a specific HBA to vfio-pci.
- Use lspci -nn to find the PCI bus ID (for example 0000:01:00.0)
- Create a file in /etc/udev/rules.d/ (for example 99-blacklist-pci.rules) with the following content to bind the device to vfio-pci and effectively disabling the default host driver:
ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:01:00.0", ATTR{driver_override}="vfio-pci"
ACTION=="add", SUBSYSTEM=="pci", KERNEL=="0000:01:00.0", RUN+="/bin/sh -c 'modprobe vfio-pci; echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind'"

 

[alex3025]

Thanks for the reply, however I had to use the following rules to make it work:

ACTION=="add", SUBSYSTEM=="pci", ENV{PCI_SLOT_NAME}=="0000:01:00.0", ATTR{driver_override}="vfio-pci"
ACTION=="add", SUBSYSTEM=="pci", ENV{PCI_SLOT_NAME}=="0000:01:00.0", RUN+="/bin/sh -c 'modprobe vfio-pci; echo 0000:01:00.0 > /sys/bus/pci/drivers/mpt3sas/unbind 2>/dev/null || true; echo 0000:01:00.0 > /sys/bus/pci/drivers/vfio-pci/bind'"

Somehow the mpt3sas driver was still getting loaded.

 

[Impact]

I recommend against blacklisting, just let VFIO take over first via softdep. You can give it specific IDs.
Are you using TrueNAS (why?) or why give a single VM control over a storage controller? Letting the node manage the storage is a better idea, IMHO.

 

[alex3025]

Yes, I'm using TrueNAS. It's the easiest way for me to have a GUI to manage shares with LDAP authentication.

The nodes manages the boot and the guests storage while the TrueNAS VM manages its own storage.