To Blacklist or Not to Blacklist...

[spetrillo]

Hello all,

I have been working on setting up my lab PVE, with PCIe passthrough and SR-IOV. I have read more articles than I can remember. Some ppl have blacklisted their physical NIC driver, some have not, and I have even seen some posts where the virtual NIC driver is blacklisted. So I open this up to those here. Do you blacklist your driver or do you not and why?

My goal is to run OPNsense as a vm with PCIe passthrough of the physical ports of my Intel I350-T4. The rest of my vms will run with SR-IOV virtual NIC functions.

Thanks,
Steve

 

[MarkusF]

Hello,

I recommend blacklisting the driver. It prevents the host from using the device as well, thus giving the VM exclusive access to the device.
https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_host_device_passthrough

 

[spetrillo]

Should I blacklist my iGPU for mdev purposes also?

 

[MarkusF]

Yes, you should blacklist the driver for the GPU and load the specific driver for enabling mdev devices.

 

[spetrillo]

Ok two more questions:

1) Should you also blacklist the virtual function driver?
2) Should we put the blacklisted drivers in the supplied pve blacklist file or create a separate one?

 

[MarkusF]

1) Should you also blacklist the virtual function driver?

I guess. What driver?
Are there separate virtual function drivers? I thought virtual functions were just a feature inside the driver/pcie-device.
If they are separate drivers (that use the device), I would blacklist them.

With lspci -nnk you can see which driver is used for a device.

2) Should we put the blacklisted drivers in the supplied pve blacklist file or create a separate one?

Your choice. You can put it in any .conf file inside /etc/modprobe.d/. I would create a new one to separate it from the pve-blacklist file.

 

[spetrillo]

Ok I think I have run into something I am not sure of. Hoping you can make sense of the following.

I created blacklist.conf, which blacklisted both my Intel NIC driver and my Intel iGPU driver. Based on the Proxmox PCI passthrough Wiki(https://pve.proxmox.com/wiki/PCI(e)_Passthrough) I should then see that the NIC and iGPU should be using vfio-pci as its driver. This did not show up after reboot. I then went and created the softdep files, for both the igb and i915 drivers. I rebooted and still no change. What am I missing?

And yes I updated Grub(update-grub) and initramfs(update-initramfs -u -k all) before rebooting each time.

 

[MarkusF]

You missed a line in the wiki:

"or the in use line is missing entirely, the device is ready to be used for passthrough."

 

[spetrillo]

So I found some additional interesting results. With pcie_acs_override=downstream not included in my Grub config I get the following result from the pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist "" command, which is the first attachment. If I add that command I get the results in the second attachment.

So the question is do I need all my I350 ports in separate groups or should they be in one common group? If I want to mdev my iGPU do I need to blacklist it also?

 

[spetrillo]

Another question on IOMMU groups. I noticed that my iGPU and onboard sound are in different IOMMU groups. I have already decided to mdev the iGPU. Should I also include the onboard sound?

 

[MarkusF]

So the question is do I need all my I350 ports in separate groups or should they be in one common group?

Separate groups if you want to assign the I350 ports to different VMs.
In general, you must have your PCIe devices in separate IOMMU groups for PCIe passthrough to be successful. This is because PCIe passthrough requires exclusive control of the device, and if devices share the same IOMMU group, they cannot be independently assigned to different virtual machines (VMs).

If I want to mdev my iGPU do I need to blacklist it also?

Like I said: Yes, if the i915 driver is being used by the host. Look at the output of lspci -nnk on the "Kernel driver in use" line.

 

[Seed]

Yes, you should blacklist the driver for the GPU and load the specific driver for enabling mdev devices.

What’s the fallout if not doing so. Say between the host and one vm that has intel iGPU not blacklisted but passed through.

 

[MarkusF]

If the GPU driver is not blacklisted on the host, both the host and the VM will try to access and control the GPU at the same time.
This will lead to conflicts.

 

[leesteken]

If the GPU driver is not blacklisted on the host, both the host and the VM will try to access and control the GPU at the same time.
This will lead to conflicts.

I feel like this has been simplified to a point where it is no longer correct anymore.

Before passing a device to a VM, it needs to use the vfio-pci driver. When starting the VM, Proxmox will automatically unbind any other driver and bind vfio-pci and reset the device. Resetting does not always work for every device, so it might be necessary to not let the actual driver bind to the device before starting the VM.

One way to achieve that is to blacklist the driver (so that it is not loaded automatically) but the drawback is that this also affects all other devices that use that same driver.

Another way it to early bind the device to vfio-pci (see the documentation). However, you need to make sure vfio-pci is loaded before the actual driver and that might require a softdep (which is unfortunately not in the documentation). This will still affect all identical devices (if you have two NICs with the same vendor:device ID, for example) but not other devices that use the same driver.