[SOLVED] Blacklist SAS2008?

[Dunuin]

Hi,

Finally moved one of my bare metal TrueNAS servers to a VM on PVE. Works fine so far and the power consumption of the PVE host only increased by 40W while I save 80W by not running an additional server, which was the main goal. The 16 disks are attached to two Dell PERC H310 flashed to IT-mode, so its basically a LSI SAS2008 HBA and I PCI passthrough those two HBAs into the VM. The HBAs reset properly and I can start/stop the TrueNAS VM as often as I like. But one annoying thing is, that PVE is seeing the disks as long as the VM isn't started yet. This screws up my external monitoring.

My idea was to blacklist the HBA drivers so PVE won't use the HBAs and can't see the disks. I tried it like blacklisting the GPU drivers for GPU passthough.
Find out what kernel modules the HBAs are using:

lspci -nk
...
03:00.0 0107: 1000:0072 (rev 03)
        Subsystem: 1028:1f1c
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
04:00.0 0107: 1000:0072 (rev 03)
        Subsystem: 1028:1f1c
        Kernel driver in use: mpt3sas
        Kernel modules: mpt3sas
...

Blacklisting it:
echo "blacklist mpt3sas" >> /etc/modprobe.d/blacklist.conf

I rebooted the server but PVE is still showing the disks attached to the HBAs.

What am I missing here?

 

[leesteken]

Blacklisting it:
echo "blacklist mpt3sas" >> /etc/modprobe.d/blacklist.conf

I rebooted the server but PVE is still showing the disks attached to the HBAs.

You need to run update-initramfs -u to apply changes to files in /etc/modprobe.d/ (and maybe update the bootloader if that does not happen automatically).

If you don't want to blacklist, you can also early bind the device to vfio-pci, but also add a softdep to make sure vfio-pci loads before mpt3sas.

 

[Dunuin]

You need to run update-initramfs -u to apply changes to files in /etc/modprobe.d/ (and maybe update the bootloader if that does not happen automatically).

If you don't want to blacklist, you can also early bind the device to vfio-pci, but also add a softdep to make sure vfio-pci loads before mpt3sas.

Thanks, that worked. What would be the benefit not blacklisting it?

@staff:
Would be great to see that in the wiki, because it isn't mentioned there.

Maybe a...

While Proxmox VE tries to deactivate the relevant drivers, blacklisting the drivers can fix issues:

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf
update-initramfs -u

and reboot your machine.

...instead of...

While Proxmox VE tries to deactivate the relevant drivers, blacklisting the drivers can fix issues:

echo "blacklist amdgpu" >> /etc/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

and reboot your machine.

 

[leesteken]

Thanks, that worked. What would be the benefit not blacklisting it?

When you have several devices that use the same driver but you don't want to passthrough all of them. Like xhci_pci or ahci or an AMD GPU for the host and one for a VM. I don't think it will make a difference for your use case.

Would be great to see that in the wiki, because it isn't mentioned there.

It is mentioned in the Proxmox manual, but not very explicitly. I assume it's a common Linux thing.