Proxmox + OPNsense + I225 NIC: only DHCP, no internet access

[TheHellSite]

Hello,

I am running an OPNsense VM on my Proxmox server. Both systems are running the latest version of their OS.
The OPNsense acts as the main router/firewall in my network providing DHCP, DNS, VPN, VLANs, ....

NICs in my Proxmox server
4 Port Intel NIC: enp5s0f0, enp5s0f1, enp6s0f0, enp6s0f1
1 Port Mainboard NIC: enp9s0

vmbr0 contains enp5s0f0 and is connected to the PPPoE port of my VDSL modem.
vmbr1 contains enp5s0f1 and is connected to the web-interface port of my VDSL modem.
vmbr2 contains all the other ports and acts as my LAN bridge.

Only the OPNsense VM is configured to use vmbr0 and vmbr1.
All other VMs / containers including the OPNsense use vmbr2 to connect to my LAN.

All physical clients that are connected to vmbr2 using enp6s0f0 or enp6s0f1 work perfectly fine.
All VMs / containers that are configured to use vmbr2 also work perfectly fine.
With "work perfectly fine" I mean that they get an IP address from my OPNsense DHCP, have internet access, VLANs are working and so on.

However any client that is connected to vmbr2 using the mainboards nic (enp9s0) of my proxmox server only has access to the DHCP server of my OPNsense and nothing else. They get a DHCP lease and that is it. They can't ping anything and are also not pingable from my OPNsense or any other client in my network.
Which doesn't make any sense to me since the OPNsense only knows that it has vmbr0, vmbr1 and vmbr2 as interfaces.
It isn't aware of the fact that vmbr2 consists out of different interfaces.

I don't know what is causing the bug my Proxmox server or my OPNsense VM.
But if I would have had to guess it is somehow related to proxmox.

Spoiler: /etc/network/interfaces

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp5s0f0 inet manual

iface enp9s0 inet manual
#Ethernet Port Mainboard

iface enp5s0f1 inet manual

iface enp6s0f0 inet manual

iface enp6s0f1 inet manual

auto vmbr0
iface vmbr0 inet manual
    bridge-ports enp5s0f0
    bridge-stp off
    bridge-fd 0
#WAN

auto vmbr1
iface vmbr1 inet manual
    bridge-ports enp5s0f1
    bridge-stp off
    bridge-fd 0
#MODEM

auto vmbr2
iface vmbr2 inet static
    address 10.10.1.250/24
    gateway 10.10.1.1
    bridge-ports enp6s0f0 enp6s0f1 enp9s0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#LAN

 

[Dunuin]

bridge-ports enp6s0f0 enp6s0f1 enp9s0

I think bridging multiple NICs may cause problems. Why do you need to bridge them at all? Wouldn't a bond0 of enp6s0f0 + enp6s0f1 connected to vmbr2 be enough?

 

[TheHellSite]

Why do you need to bridge them at all?

Because I have 3 switches going to 3 different locations hooked up to these ports.

Wouldn't a bond0 of enp6s0f0 + enp6s0f1 connected to vmbr2 be enough?

You mean to first bond enp6s0f0 + enp6s0f1 and then add bond + enp9 to the bridge?

Whats the difference between bonding and bridging?

 

[Dunuin]

Because I have 3 switches going to 3 different locations hooked up to these ports.

Why you dont just daisy chain the switches? One NIC connected from Proxmox to switch A and switch A connected to switch B and switch C. That way you dont need 3 NICs and you would get some unused NICs for network separation (DMZ) or redundancy.

You mean to first bond enp6s0f0 + enp6s0f1 and then add bond + enp9 to the bridge?

Whats the difference between bonding and bridging?

With bridging the NICs just work like a unmanaged switch. But slower and less ressource efficient than just using a physical hardware switch.
With bonding your NICs can team up and double the throughput (if your switch supports that... LACP for example... so you get 2Gbit) and you get redundancy so if one of the 2 bonded NICs is failing the connection is still up.

 

[TheHellSite]

Why you dont just daisy chain the switches? One NIC connected from Proxmox to switch A and switch A connected to switch B and switch C. That way you dont need 3 NICs and you would get some unused NICs for network separation (DMZ) or redundancy.

Sure, I could do this but this wouldn't solve the issue, it would only be a workaround!
For network seperation I configured VLANs.

Don't get me wrong! I appreciate your advise but there is nothing totally wrong with my setup.
I think there is a bug in either proxmox or opnsense.

The question know is: Is it possible to bridge ports from different NICs on one vmbr?

 

[TheHellSite]

No one here that could help?

 

[TheHellSite]

SOLVED!
https://forum.proxmox.com/threads/fixing-connectivity-issue-on-i225-v-nic-workarounds.80226/

Fix: https://forum.proxmox.com/threads/kernel-5-11.86225/