To Blacklist or Not to Blacklist...

[spetrillo]

Hello all,

I have been working on setting up my lab PVE, with PCIe passthrough and SR-IOV. I have read more articles than I can remember. Some ppl have blacklisted their physical NIC driver, some have not, and I have even seen some posts where the virtual NIC driver is blacklisted. So I open this up to those here. Do you blacklist your driver or do you not and why?

My goal is to run OPNsense as a vm with PCIe passthrough of the physical ports of my Intel I350-T4. The rest of my vms will run with SR-IOV virtual NIC functions.

Thanks,
Steve

 

[MarkusF]

Hello,

I recommend blacklisting the driver. It prevents the host from using the device as well, thus giving the VM exclusive access to the device.
https://pve.proxmox.com/wiki/PCI(e)_Passthrough#_host_device_passthrough

 

[spetrillo]

Should I blacklist my iGPU for mdev purposes also?

 

[MarkusF]

Yes, you should blacklist the driver for the GPU and load the specific driver for enabling mdev devices.

 

[spetrillo]

Ok two more questions:

1) Should you also blacklist the virtual function driver?
2) Should we put the blacklisted drivers in the supplied pve blacklist file or create a separate one?

 

[MarkusF]

1) Should you also blacklist the virtual function driver?

I guess. What driver?
Are there separate virtual function drivers? I thought virtual functions were just a feature inside the driver/pcie-device.
If they are separate drivers (that use the device), I would blacklist them.

With lspci -nnk you can see which driver is used for a device.

2) Should we put the blacklisted drivers in the supplied pve blacklist file or create a separate one?

Your choice. You can put it in any .conf file inside /etc/modprobe.d/. I would create a new one to separate it from the pve-blacklist file.

 

[spetrillo]

Ok I think I have run into something I am not sure of. Hoping you can make sense of the following.

I created blacklist.conf, which blacklisted both my Intel NIC driver and my Intel iGPU driver. Based on the Proxmox PCI passthrough Wiki(https://pve.proxmox.com/wiki/PCI(e)_Passthrough) I should then see that the NIC and iGPU should be using vfio-pci as its driver. This did not show up after reboot. I then went and created the softdep files, for both the igb and i915 drivers. I rebooted and still no change. What am I missing?

And yes I updated Grub(update-grub) and initramfs(update-initramfs -u -k all) before rebooting each time.

 

[MarkusF]

You missed a line in the wiki:

"or the in use line is missing entirely, the device is ready to be used for passthrough."

 

[spetrillo]

So I found some additional interesting results. With pcie_acs_override=downstream not included in my Grub config I get the following result from the pvesh get /nodes/{nodename}/hardware/pci --pci-class-blacklist "" command, which is the first attachment. If I add that command I get the results in the second attachment.

So the question is do I need all my I350 ports in separate groups or should they be in one common group? If I want to mdev my iGPU do I need to blacklist it also?

 

[spetrillo]

Another question on IOMMU groups. I noticed that my iGPU and onboard sound are in different IOMMU groups. I have already decided to mdev the iGPU. Should I also include the onboard sound?

 

[MarkusF]

So the question is do I need all my I350 ports in separate groups or should they be in one common group?

Separate groups if you want to assign the I350 ports to different VMs.
In general, you must have your PCIe devices in separate IOMMU groups for PCIe passthrough to be successful. This is because PCIe passthrough requires exclusive control of the device, and if devices share the same IOMMU group, they cannot be independently assigned to different virtual machines (VMs).

If I want to mdev my iGPU do I need to blacklist it also?

Like I said: Yes, if the i915 driver is being used by the host. Look at the output of lspci -nnk on the "Kernel driver in use" line.

 

[Seed]

Yes, you should blacklist the driver for the GPU and load the specific driver for enabling mdev devices.

What’s the fallout if not doing so. Say between the host and one vm that has intel iGPU not blacklisted but passed through.

 

[MarkusF]

If the GPU driver is not blacklisted on the host, both the host and the VM will try to access and control the GPU at the same time.
This will lead to conflicts.

 

[leesteken]

If the GPU driver is not blacklisted on the host, both the host and the VM will try to access and control the GPU at the same time.
This will lead to conflicts.

I feel like this has been simplified to a point where it is no longer correct anymore.

Before passing a device to a VM, it needs to use the vfio-pci driver. When starting the VM, Proxmox will automatically unbind any other driver and bind vfio-pci and reset the device. Resetting does not always work for every device, so it might be necessary to not let the actual driver bind to the device before starting the VM.

One way to achieve that is to blacklist the driver (so that it is not loaded automatically) but the drawback is that this also affects all other devices that use that same driver.

Another way it to early bind the device to vfio-pci (see the documentation). However, you need to make sure vfio-pci is loaded before the actual driver and that might require a softdep (which is unfortunately not in the documentation). This will still affect all identical devices (if you have two NICs with the same vendor:device ID, for example) but not other devices that use the same driver.

 

[jemme]

Hello (sorry to revive the thread),

I'm trying to pass my GPU (NVIDIA 3060) to a VM which runs Windows11 using a physical NVMe SSD as the disk (basically a virtualization of an already existing disk). The problem is that I also have a Quadro P400 using the same drives as the 3060, so I can't blacklist the drivers since they both use the "nouveau" drivers as seen when running lspci -nnk

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
        Subsystem: NVIDIA Corporation GP107GL [Quadro P400] [10de:11be]
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
        Subsystem: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:11be]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA106 [GeForce RTX 3060] [10de:2503] (rev a1)
        Subsystem: ASUSTeK Computer Inc. GA106 [GeForce RTX 3060] [1043:87f3]
        Kernel driver in use: nouveau
        Kernel modules: nvidiafb, nouveau
07:00.1 Audio device [0403]: NVIDIA Corporation GA106 High Definition Audio Controller [10de:228e] (rev a1)
        Subsystem: ASUSTeK Computer Inc. GA106 High Definition Audio Controller [1043:87f3]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

Last time I passed the 3060, it crashed the Proxmox install and hence the VM too (The GUI for the node stopped working and the VM was unresponsive but outputting an image when a HDMI cable was plugged into the card). I since then reinstalled Proxmox baremetal since I was just trying to do this with the node and had nothing important on it. At first, I thought that the problem was in the IOMMU grouping, but now that I read @leesteken answer I think it was the blacklisting.

One way to achieve that is to blacklist the driver (so that it is not loaded automatically) but the drawback is that this also affects all other devices that use that same driver.

Just in case it is the IOMMU groups, heres the output of pvesh get /nodes/(node name)/hardware/pci --pci-class-blacklist ""

┌──────────┬────────┬──────────────┬────────────┬────────┬───────────────────────────────────────────────────────
│ class    │ device │ id           │ iommugroup │ vendor │ device_name                                          
╞══════════╪════════╪══════════════╪════════════╪════════╪═══════════════════════════════════════════════════════
│ 0x010601 │ 0xa382 │ 0000:00:17.0 │          4 │ 0x8086 │ 400 Series Chipset Family SATA AHCI Controller      
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x010802 │ 0x5006 │ 0000:03:00.0 │          5 │ 0x15b7 │ WD Black SN750 / PC SN730 NVMe SSD                  
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x010802 │ 0x1202 │ 0000:08:00.0 │          7 │ 0x1e4b │ NVMe SSD Controller MAP1202                          
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x020000 │ 0x8168 │ 0000:06:00.0 │          6 │ 0x10ec │ RTL8111/8168/8411 PCI Express Gigabit Ethernet Control
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x028000 │ 0x2723 │ 0000:02:00.0 │          5 │ 0x8086 │ Wi-Fi 6 AX200                                        
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x030000 │ 0x1cb3 │ 0000:01:00.0 │          1 │ 0x10de │ GP107GL [Quadro P400]                                
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x030000 │ 0x2503 │ 0000:07:00.0 │          6 │ 0x10de │ GA106 [GeForce RTX 3060]                            
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x040300 │ 0xa3f0 │ 0000:00:1f.3 │          8 │ 0x8086 │ Comet Lake PCH-V cAVS                                
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x040300 │ 0x0fb9 │ 0000:01:00.1 │          1 │ 0x10de │ GP107GL High Definition Audio Controller            
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x040300 │ 0x228e │ 0000:07:00.1 │          6 │ 0x10de │ GA106 High Definition Audio Controller              
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x058000 │ 0xa3a1 │ 0000:00:1f.2 │          8 │ 0x8086 │ Cannon Lake PCH Power Management Controller          
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060000 │ 0x9b43 │ 0000:00:00.0 │          0 │ 0x8086 │ 10th Gen Core Processor Host Bridge/DRAM Registers  
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060100 │ 0xa3c8 │ 0000:00:1f.0 │          8 │ 0x8086 │ B460 Chipset LPC/eSPI Controller                    
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0x1901 │ 0000:00:01.0 │          1 │ 0x8086 │ 6th-10th Gen Core Processor PCIe Controller (x16)    
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0xa3e9 │ 0000:00:1b.0 │          5 │ 0x8086 │                                                      
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0xa3eb │ 0000:00:1b.4 │          5 │ 0x8086 │ Comet Lake PCI Express Root Port #21                
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0xa392 │ 0000:00:1c.0 │          6 │ 0x8086 │                                                      
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0xa393 │ 0000:00:1c.3 │          6 │ 0x8086 │                                                      
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0xa394 │ 0000:00:1c.4 │          6 │ 0x8086 │ Comet Lake PCI Express Root Port #05                
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0xa398 │ 0000:00:1d.0 │          7 │ 0x8086 │ Comet Lake PCI Express Root Port 9                  
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x060400 │ 0x1080 │ 0000:04:00.0 │          6 │ 0x1b21 │ ASM1083/1085 PCIe to PCI Bridge                      
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x078000 │ 0xa3ba │ 0000:00:16.0 │          3 │ 0x8086 │ Comet Lake PCH-V HECI Controller                    
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x0c0330 │ 0xa3af │ 0000:00:14.0 │          2 │ 0x8086 │ Comet Lake PCH-V USB Controller                      
├──────────┼────────┼──────────────┼────────────┼────────┼───────────────────────────────────────────────────────
│ 0x0c0500 │ 0xa3a3 │ 0000:00:1f.4 │          8 │ 0x8086 │ Comet Lake PCH-V SMBus Host Controller              
└──────────┴────────┴──────────────┴────────────┴────────┴───────────────────────────────────────────────────────

... Now that I look at it properly, why is there some blank spaces for the 6th group for the 3060?

Anyways, I haven't passed it yet since I've been doing this none stop since this morning and now its late lol. Thank you for the help in advance!

EDIT: Tried to pass it using the 6th group and without blacklisting, same problem/crash happens

 

[spetrillo]

So let me get this straight....

When you try to passthrough either GPU the VM hangs or crashes and then the PVE hangs up? Did I get that right? Are you running LVM or ZFS? Can you show us yourGrub or Cmdline config?

Originally when trying to passthrough my I350 Intel PCI adapter my vm would die and then the PVE would hang completely. I have boiled down my config to the following:

1) Update of /etc/kernel/cmdline to: intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction (you might not need the pcie_acs_override, as that is for getting devices in the their own IOMMU groups)
2) Update of /etc/modules to:
vfio
vfio_iommu_type1
vfio_pci

Your mileage may vary but this works for me on my Lenovo M720q Tiny PC. I can now passthrough all 4 ports of the Intel adapter to my VM. A couple things about the vm config. You must use Q35 as the machine type and should use UEFI for your BIOS config. I would assume you are adding the GPU as a PCI device? Are you checking ROM-Bar and PCI-Express?

 

[jemme]

Thank you so much for taking the time for helping me!!!

The problem is that when I start the VM, the webGUI can't connect to the node and considers it like offline. When I plug a HDMI cable to the GPU, it outputs an image, but the keyboard and mouse doesn't work. I just noticed that my RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller is in the same group as the GPU. I also read about passing the USB devices to the VM in addition to the GPU. So I will try that and see what happens to see if its the Ethernet Controller that makes it seem like it hangs.

To answer your questions:

Are you running LVM or ZFS? Can you show us yourGrub or Cmdline config?

My EFI and TPM are on the default local-lvm storage. This is what I get when i run proxmox-boot-tool status

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace...
System currently booted with uefi
4723-10A7 is configured with: grub (versions: 6.8.12-4-pve, 6.8.12-5-pve)

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt" is in my grub config file. My /etc/modules looks like this, which is similar to yours

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

When I add the GPU, I actually pass the GPU and the audio as two different PCI devices. The GPU has PCI express, ROM-bar and Primary GPU checked. The audio has ROM-bar and PCI expressed checked. They both use the Device's ROM-bar.

UPDATE: I passed the USBs and it works! The VM runs!!! But the connection to the server is lost. The reason is exactly as I thought: the Ethernet wired connection is deactivated (I can see it happen since the blinking light stops the moment I run the VM after a reboot). Upon closing the VM, the PC still runs but the connection to the server doesn't come back (since it got "passed-through"). Other problem: the VM doesn't have internet. Maybe it is because it is set to Intel E1000? I'm not sure of what all the options mean so I left it like that.

 

[spetrillo]

Are you trying to pass the Quadro or the GeForce? If its the GeForce then you have some problems bc it is not in its own IOMMU group and could be your issue.

So lets talk about your PC. What kind of PC is it? How many PCIe slots does it have? Which PCIe slots are the GPUs plugged into? Have you tried to pass one GPU and removed the other one?

 

[jemme]

Thank you for the reply! Let me answer you

Are you trying to pass the Quadro or the GeForce? If its the GeForce then you have some problems bc it is not in its own IOMMU group and could be your issue.

I'm trying to pass the GeForce, so that maybe be why I'm having trouble with it... I would prefer not doing the acs_override so I might switch the card places in the end, but before that I'll try to see if the problem is with the ethernet port being named wrongly like seen in this reddit post.

So lets talk about your PC. What kind of PC is it? How many PCIe slots does it have? Which PCIe slots are the GPUs plugged into? Have you tried to pass one GPU and removed the other one?

I do love talking about my PC )))))) Ok so the motherboard is a Asus PRIME B460-PLUS ATX with a i7-10700K with a GeForce 3060. It was an gaming PC at first, but I didn't use it a lot so I'm converting it into a little server with a Remote Gaming VM hence all these posts. I added the Quadro for a future Jellyfin media library to do video transcoding on different LXC which I will do after the pass-through since that is much easier (I think). It had a WIFI PCI adapter on the closest PCIe slot of the CPU, but I recently removed it since I thought that was the problem. Just to clarify, from the CPU socket to the bottom of the board:

1. Wifi (Gigabyte GC-WBAX200) in a PCI slot (I think)
2. Quadro P400 on the 16x PCIe slot (I think)
3. all empty until the second 16x PCIe slot
4. GeForce 3060 on the 16x PCIe slot (if the Quadro is on the 16x slot, then this one is on the one that can run in 4x mode (if that changes anything))

Looking back at the IOMMU grouping, I should swap the GPUs since the Quadro is in its own group. Before installing Proxmox on it, I had the GeForce on the first PCIe 16x slot, but PCPartPicker recommended I put the Quadro on top. I haven't tried passing with only one GPU on the motherboard, but I will try if it comes to it.

UPDATE: The reddit post didn't help since the networking is right. @spetrillo is right, there's something else in the IOMMU group of the GeForce making it fail. I'm removing the GPU pass-through, swapping the cards and then passing it again to see if it works, I'll let yall know if it works and I'll post my set up

 

[jemme]

Final message: IT WORKED!!! Swapping the GPUs was the key to success!

When running the following command found on this post (for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done), it was evident that the group of the GeForce contained the Ethernet port. When the GPU was passed to the VM, it seemed like the server was down but the connection was only lost. After removing the line options vfio-pci ids=(id of GPU),(id of audio) disable_vga=1 in /etc/modprobe.d/vfio.conf (just to be safe), removing the PCI device from the VM in the Hardware tab of the webGUI, and swapping the cards' PCIe slots, the command showed that the GeForce was alone in its IOMMU group like needed for this to work. I then re-did the process in reverse and started the VM. Everything worked like intended!

Note: The Quadro is now in the same IOMMU group as the Ethernet Port, which means that if it is to be passed to something, I will have the same problem... But looking at this video that I previously linked, I don't think I will run into that problem. I also noticed that I have a USB controller in a IOMMU group that I could pass instead of one USB device at a time (mouse and keyboard). I will pass it instead since I want to plug/unplug a DDR pad to my computer whenever my roommate uses it.

Note-again: I'm thinking of posting my entire experience but I don't know where it would be appropriate to do so since it is irrelevant to this thread. This whole saga is so my roommate can use the computer for a modded DDR plateform while I can still use the computer for schoolwork/gaming (It's his Christmas gift lol, don't think too much about it hahahah). Does someone know the best place/way to share it?