[SOLVED] Proxmox multiple interfaces without bonding in one vmbr useable?

[frozeneye]

Hi,

hopefully somebody can help here...

We use a firewall appliance (intel x86-Hardware) as a proxmox host. This appliance has a lot of network interfaces. Idea is to use some of the ports like a switch.

Details:

• Proxmox 8 latest version
• Opensense Firewall
• multiple network segmentations provided by Opensense Firewall
• some WRT-flashed Accesspoints

It was easy to get Proxmox and Opensense up and running. We use vmbr0 with one interface as Uplink to the internet and first management IP from proxmox.

We build a vmbr1 VLAN aware with 3 network Interfaces added (no bonding!)

Then we build Opensense with some interfaces (1x WAN to vmbr0, 3x LAN (2 with different VLAN Tags (10,20) to vmbr1, one without VLAN). We provide 3 LAN-Interfaces in Opensense with own IPs and DHCPv4-service. So we were able to use the untagged network (for Opensense management and routed Proxmox management also) and both tagged networks for 2 Client networks from vmbr1.

Everything was working as expected to this point. We have connect one Interface from vmbr1 directly to a computer and we got access to all 3 networks (by using VLAN tagging on this device).

Now we want to use two wrt-based WLAN accesspoints on the other interfaces in vmbr1 to get VLAN-tagged WLANs, each Access-Point should open 2 SSIDs and connect each one to one VLAN.

Only Proxmox "knows" the VLAN Tags and the external devices also.

Here we have problems. we get some some connections, but f.eg. no DHCP-IP on the WLAN Client. It seems, that the network interfaces in vmbr1 doesn't act as expected. When more than one interface in this vmbr is active (=has a device connected), we have problems.

Is it possible, that a vmbr with multiple interfaces without any bonding or LACP is not able to "route" packets back to the same interface like a vSwitch in ESXi (Route based on IP Hash or others)? We don't want to use a switch here if possible, its a special project for a small nonprofit org...

Following the simplified network config

auto lo
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

iface eno5 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.x.20/24
        gateway 192.168.x.1
        bridge-ports eno2
        bridge-stp off
        bridge-fd 0
        
auto vmbr1
iface vmbr1 inet static
        address 192.168.y.20/24
        bridge-ports eno3 eno4 eno5
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Thanks for your help!

 

[weehooey-bh]

Hi, fundamentally, what you are trying to do is possible.

Please post the configuration for your OPNsense VM. You will find it here: /etc/pve/nodes/<node>/qemu-server/

Side question: Is there a reason you have your PVE node with IP addresses on both bridges?

 

[frozeneye]

Hi,

thanks for your reply and sorry for being late.

Side answer: Later on Proxmox should be available via second IP behind the Opensense, the first IP will be in place as a fallback.

Here is the simplified conf from the opensense:

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
efidisk0: data_sata:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/OPNsense-24.1-dvd-amd64.iso,media=cdrom,size=1936730K
memory: 4096
meta: creation-qemu=8.1.5,ctime=1717850899
name: opensense01
net0: virtio=BC:xxxx,bridge=vmbr0
net1: virtio=BC:xxxx,bridge=vmbr1,firewall=1,tag=10
net2: virtio=BC:xxxx,bridge=vmbr1,firewall=1,tag=20
net3: virtio=BC:xxxx,bridge=vmbr1
numa: 0
ostype: l26
scsi0: data_sata:vm-100-disk-1,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=xxxx
sockets: 1
vmgenid: xxxx

Just to clarify: first basic tests with only one connected external device works. When more than one external device are connected, we have the problems.
Thanks!

 

[weehooey-bh]

Hi,

thanks for your reply and sorry for being late.

Side answer: Later on Proxmox should be available via second IP behind the Opensense, the first IP will be in place as a fallback.

Here is the simplified conf from the opensense:

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 2
cpu: x86-64-v2-AES
efidisk0: data_sata:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
ide2: local:iso/OPNsense-24.1-dvd-amd64.iso,media=cdrom,size=1936730K
memory: 4096
meta: creation-qemu=8.1.5,ctime=1717850899
name: opensense01
net0: virtio=BC:xxxx,bridge=vmbr0
net1: virtio=BC:xxxx,bridge=vmbr1,firewall=1,tag=10
net2: virtio=BC:xxxx,bridge=vmbr1,firewall=1,tag=20
net3: virtio=BC:xxxx,bridge=vmbr1
numa: 0
ostype: l26
scsi0: data_sata:vm-100-disk-1,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=xxxx
sockets: 1
vmgenid: xxxx

Just to clarify: first basic tests with only one connected external device works. When more than one external device are connected, we have the problems.
Thanks!

Thanks for sharing the config and the side answer

Can you tell me more about the testing and what you see when things fail?

Have you tested with two computers as devices on eno3, eno4 and eno5? Or just with a computer and one of the APs?

If only connecting one AP, are your tests successful with two computers connected to the single AP?

Have you done packet captures on the OPNsense? The PVE host? The devices connected to the APs?

In OPNsense, what are you seeing on the ARP table? If using IPv6, on NDP table?

How are you testing for connectivity? Is it only with successful DHCP address assignment? Is OPNsense the DHCP server? Have you tried with static IPs and ping?

I am wondering if you are getting a loop somewhere or there is an issue with the configuration of the APs.

 

[frozeneye]

On my first basic test I've connected a macBook to one port and add VLANs on it. So i get all 3 networks (untagged and both tagged VLANs) connected and on each network DHCP via openSense works fine. all other network ports are not used during this first test.

When I connect another device ins this vmbr, f.eg. a WLAN AP, some weird things are starting. I try to connect f.eg. from the wired connected notebook to a Web-IF from a device inside one of the opensense-LANs and the site hangs (but it will be initially connected). Ping works fine. When i move the device to another vmbr with one interface and connect to that wone, everything is fine with and without VLAN.

It seems that the request works and the answer (or parts of it) doesn't find the back route.

ATM I don't have it inspected deeper. Its possible that the WLAN APs are part of this problem. I will try to clarify that when I'm able to get back to that project...

 

[weehooey-bh]

Interesting. Once you dig a little further, please update.

If a ping works and an HTTP connection starts but fails, I would focus on the OPNsense VM and work from there. Stateful firewalls can behave that way with asymmetric routing — not necessarily your issue.

For testing, I would remove the host IP address from vmbr1 only leaving an IP address on vmbr0.

 

[frozeneye]

Hi, sorry for being late here....

Problem is solved. Weird configs on WLAN-AP (Openwrt-based) fixed and everything works as expected.WLAN-AP-config wasn't in my scope.

Thanks for your help!