Unable to create bridge

[sunseeker2k5]

Good day ! Nice to virtually meet you all. I have been running Proxmox for 2 years now on various hardware. While I am no expert, I know my way around the system.
I am on 8.2.7 using kernel 6.8.12-3. The original install was Proxmox 8 when it came out. System was updated to the current version in November 2024.
Proxmox runs on a mini PC (Lenovo P330) which has its onboard nIC (en0) which i use for the proxmox bridge (vmbr0).
It also has a quad port Intel Gigabit card (HP NC365T).
The first 2 ports are passed onto PFSENSE via PCI pass-thru and they work perfectly as LAN and WAN port (enp1s0f0 and enp1s0f1)
The other 2 ports have not been used ever.

I want to add a new bridge using one of the unused ports on the HP NC365T.(enp1s0f2 or enp1s0f3)
1.LSPCI shows all 4 ports but ip link and addr checks only show en0 and the existing bridge vmbr0. If i check interfaces in terminal it sees all 4 of them (enp1s0f0-enp1s0f3)
2-5 : I proceed to add the bridge and I get an error that enp1s0f3 doesn't exist.

I can add and use enp1s0f2 or enp1s0f3 to a VM guest (by passing them thru) and they work perfectly. It's just for some reason I cannot add them to a proxmox bridge.

I attached a screenshot to show what I see and what i did. Any input is appreciated. Thank you.

 

[sunseeker2k5]

I did some checks on a different system (Asrock Steel Legend B660 and I5-12400), same quad network card but on Proxmox 7.2.11 and kernel 5.15.64-1
- The not passed thru ports from the quad NIC are visible in ip addr checks
- I can create the bridge without issues

Is this kernel or hardware related then ? I am puzzled, your input would be apprciated.

 

[gfngfn256]

You don't provide details of how you accomplished the passthrough.

I did some checks on a different system

Different M/B BIOSs etc. accomplish passthrough differently.

Also AFAIK, whether or not you can split the passthrough of the different ports on the same card will be both specific card-dependent & PCI slot MB & CPU dependent. You will need to check the IOMMU groupings for correct IOMMU isolation - as shown here in the Proxmox wiki.

You could use the following for a full IOMMU listing with ports etc.

for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

I'm not sure why you chose to use any passthrough at all. Why not just assign different bridges to each port (vmbr1 for enp1s0f0, vmbr2 for enp1s0f1 etc.) & then give the various VMs the vNICs on those bridges; so your pfSense VM would be given vmbr1 & vmbr2 for wan & lan & other VMs could be given by example vmbr3 . You may have had your reasons for this, but IDK.

 

[sunseeker2k5]

Thank you @gfngfn256 for taking the time to review and respond.

This is how I did the pass thru of the 2 ports :
I enabled IOMMU by
1.VT-D is on in the BIOS.

2.Add this in GRUB : GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"

3.updated /etc/modules with
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

4.rebooted and checked if IOMMU was functional
root@pve:~# dmesg | grep -e DMAR -e IOMMU
[ 0.012091] ACPI: DMAR 0x000000009AF087A0 0000A8 (v01 LENOVO TC-M1U 00001720 01000013)
[ 0.012155] ACPI: Reserving DMAR table memory at [mem 0x9af087a0-0x9af08847]
[ 0.040195] DMAR: IOMMU enabled
[ 0.114141] DMAR: Host address width 39
[ 0.114142] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.114154] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.114157] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.114162] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.114165] DMAR: RMRR base: 0x00000099c31000 end: 0x00000099c50fff
[ 0.114167] DMAR: RMRR base: 0x0000009d000000 end: 0x0000009f7fffff
[ 0.114170] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.114172] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.114174] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.116666] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.491395] DMAR: No ATSR found
[ 0.491397] DMAR: No SATC found
[ 0.491399] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.491400] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.491402] DMAR: IOMMU feature nwfs inconsistent
[ 0.491404] DMAR: IOMMU feature pasid inconsistent
[ 0.491405] DMAR: IOMMU feature eafs inconsistent
[ 0.491407] DMAR: IOMMU feature prs inconsistent
[ 0.491408] DMAR: IOMMU feature nest inconsistent
[ 0.491409] DMAR: IOMMU feature mts inconsistent
[ 0.491411] DMAR: IOMMU feature sc_support inconsistent
[ 0.491412] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.491414] DMAR: dmar0: Using Queued invalidation
[ 0.491418] DMAR: dmar1: Using Queued invalidation
[ 0.493784] DMAR: Intel(R) Virtualization Technology for Directed I/O

5.Passed 2 ports thru to PFSENSE as below :


IOMMU groups are below (per your command) :
IOMMU group 0 00:02.0 VGA compatible controller [0300]: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] [8086:3e92]
IOMMU group 10 02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
IOMMU group 11 03:00.0 Non-Volatile memory controller [0108]: Shenzhen Unionmemory Information System Ltd. NVMe 256G SSD device [1cc4:17ab]
IOMMU group 1 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec2] (rev 07)
IOMMU group 2 00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 2 01:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 2 01:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 2 01:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 2 01:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 3 00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU group 4 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU group 4 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU group 5 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU group 5 00:16.3 Serial controller [0700]: Intel Corporation Cannon Lake PCH Active Management Technology - SOL [8086:a363] (rev 10)
IOMMU group 6 00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10)
IOMMU group 7 00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 [8086:a340] (rev f0)
IOMMU group 8 00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 [8086:a32c] (rev f0)
IOMMU group 9 00:1f.0 ISA bridge [0601]: Intel Corporation Q370 Chipset LPC/eSPI Controller [8086:a306] (rev 10)
IOMMU group 9 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU group 9 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU group 9 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-LM [8086:15bb] (rev 10)

The quad gig nic ports are in the same IOMMU group (Group 2). Would this be the reason ? If so, why can i pass the 2 free ports into other VMS without issues ?

As to why I passed thru the 2 ports to PFSENSE ?
- smaller attack surface (whole hypervisor exposed vs PFSENSE only)
- I wasn't comfortable securing the Hypervisor side when exposed to the Internet (Proxmox firewall), while I was comfortable with PFsense
- updates (updating PFSENSE only and rolling back if there are issues are easier vs updating hypervisor and rolling back.)
- speed (I had better results saturating my symmetric 1G Internet when passing thru)
- when having passed thru NICs to PFSENSE, I am not impacted by potential changes to predictable network interface names (like what happened when upgrading to proxmox 8 from 7). At the time I didn't know I could assign static names to the Interfaces and I haven't updated the system yet.

I hope it makes sense.

Edit : Checking the other system where it works ok, each port has its own IOMMU group :
IOMMU group 12 01:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 13 01:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 14 01:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU group 15 01:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)