[SOLVED] PCIe passthrough with more than one NIC don't work

[BurningSquirrel-de]

Hello forum,

what I want to do, is passthrough three pcie network cards to one virtual machine to act as a OPNsense firewall.

I'm runnig a fresh installed Proxmox VE 6.4, passthrough is exactly configured like discribe in the Proxmox Doc (https://pve.proxmox.com/wiki/Pci_passthrough).
I create a new guest in the Web GUI (the only special is that I use a q35 machine and UEFI BIOS), choose no network card in the assistant and when the machine is created I add the network pcie cards, by adding a new hardware, pci device, choose card and check pci-express.

When i add one card the guest start, when i add more than one, the whole Proxmox VE Web GUI crashes and for sure the guest won't start. I can reboot the Bare Metal when i logon directly in the shell.

I followed the problem back to the point that i think it's the problem -> IOMMU groups.
All the NICs are in one IOMMU group and in the most docs and tutorials its said more or less "For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM." But how can i bring the devices in seperate groups?

I ask Google for help an allready try these things:

• Add "pcie_acs_override=downstream,multifunction" to boot cmdline in grub
• Fresh install and insert on NICard after the other

Has someone an idea that could help me? I'm well experienced in IT, but i am not very familliar with problems in virtualization (everything worked fine till to this computer), what i want to say, i do not want to rule out, that i not see a very basic mistake.

Thanks for any idea or help with this problem in advance.

 

[leesteken]

The IOMMU groups are fully determined by your motherboard BIOS, based on actual device isolation of the hardware on it. That is also why all devices in a group need to go to the same VM (or all to the host): otherwise they won't be properly isolated.
The pcie_acs_override allows you to ignore this, but you lose the guarantee that devices cannot influence each other and that is a security issue if you untrusted VMs for other people.
There is also never a guarantee that the PCI(e) devices will work with passthrough, even when you ignore/break-up the IOMMU groups. Can you show us your groups (without using pcie_acs_override) using this command? for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done
Maybe you only need to break the group of the multi-NIC-device (or multiple NIC-devices?) and we can see what devices you used and if they are known to work or to have problems with this.

 

[BurningSquirrel-de]

Hello avw,

many thanks for your answer and your explanation.

The command you told me to enter, outputs the following:

IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:9b53] (rev 03)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 03)
IOMMU group 1 01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8161] (rev 15)
IOMMU group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Device [8086:9bc8] (rev 03)
IOMMU group 3 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a3af]
IOMMU group 4 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a3ba]
IOMMU group 5 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a382]
IOMMU group 6 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a3e9] (rev f0)
IOMMU group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a392] (rev f0)
IOMMU group 7 00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:a393] (rev f0)
IOMMU group 7 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a394] (rev f0)
IOMMU group 7 00:1c.5 PCI bridge [0604]: Intel Corporation Device [8086:a395] (rev f0)
IOMMU group 7 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU group 7 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8161] (rev 15)
IOMMU group 7 06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. Device [10ec:8161] (rev 15)
IOMMU group 8 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a398] (rev f0)
IOMMU group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a3c8]
IOMMU group 9 00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a3a1]
IOMMU group 9 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a3a3]

I think the device (on) 01:00.0 is one NIC that is insert in the x16 PCIe port on the mainboard, 04:00.0 is the onboard NIC and 05:00.0 and 06:00.0 are the NICs in the x1 PCIe ports.

01, 05 and 06 should be dedicated to the OPNsense firewall, 04 should be a bridged port shared by a VM (only a light Windows machine to monitor the network) and for the WebGUI of Proxmox.

 

[leesteken]

Your guess might be correct: the device plugged into the x16 slot is separate from the device plugged into slot that are probably connected to the motherboard chipset. However, given these IOMMU groups, why not pass 04:00.0, 05:00.0 and 06:00.0 to the OPNsense firewall (because they are all in the same group 7) and pass 01:00.0 (which is alone in a group 1) to the other VM. This would match the device isolation and security (if you did not use pcie_acs_override, which I don't think you did because of the shown grouping) and all should work fine. If you insist on switching 01:00.0 and 04:00.0, it might not work and definitely won't be secure. Please just switch the cables and match the IOMMU groups, it could save your a lot of problems.

 

[BurningSquirrel-de]

Hello!

Thanks for your help! Going this way of solution is obviously the best, I tried it and it worked. I think i was to fixed on a solution rearrange the groups that i don't see this obviously way.