SDN Broadcasting? (What are best practices)

[tsumaru720]

Hey all

Previously my networking was as follows

vmbr0 attached to eth1 - vmbr0 used for management IP. Management vlan is untagged.

vmbr110 attached to eth1.110 - no ip on host
vmbr120 attached to eth1.120 - no ip on host
vmbr130 attached to eth1.130 - no ip on host
etc to use for guest networking as appropriate.

These vmbr devices were all configured in /etc/network/interfaces.

I've installed and set up the SDN packages to try to simplify my network management moving forward...

1. I've removed all the extra vmbr11x and eth1.xx devices from /etc/network/interfaces
2. I've made vmbr0 vlan aware
3. I've added a single (vlan) zone to proxmox, using vmbr0 as the bridge
4. I've added new vnets using sdn interface - vlan110 with a tag of 110, vlan120 with a tag of 120 etc
5. moved all my guest networks to the new interfaces

Everything looks fine, except my VMs that sit in my infrastructure/management VLAN have their network set to be vmbr0 - as a result they seem to be seeing all traffic on that bridge, regardless of the vlan tag... its like the linux bridge doesnt know where to forward to so its broadcasting

# brctl show
bridge name     bridge id               STP enabled     interfaces
vlan110         8000.00221965528e       no              tap143i0
                                                        veth138i0
                                                        vmbr0.110
vlan120         8000.00221965528e       no              veth274i0
                                                        veth277i0
                                                        vmbr0.120
vlan130         8000.00221965528e       no              vmbr0.130
vlan150         8000.00221965528e       no              tap501i0
                                                        tap9901i1
                                                        vmbr0.150
vlan999         8000.00221965528e       no              tap9901i0
                                                        vmbr0.999
vmbr0           8000.00221965528e       no              eth1
                                                        tap901i0

Has anyone seen this kind of behaviour before and/or know how to fix it?

 

[Hannes Laimer]

Hey,

it's a bit unusual, as normally the Linux bridge should be handling the VLAN tagging and untagging correctly. Here are a couple of steps to troubleshoot:

1. Double-check your network configuration: First, please make sure that your network configuration is correct. In particular, ensure that the VLAN IDs for each of your vnets match their respective VLAN tags. Also, check that you've correctly configured the "vlan-aware" setting on vmbr0.
2. Check for MAC address conflicts: The output from brctl show shows that all your bridges have the same bridge id (8000.00221965528e). This could potentially cause issues if multiple bridges are trying to handle traffic for the same MAC address. Please check to ensure that there are no MAC address conflicts on your network.
3. Check for VLAN configuration on your switch: If you have a managed switch, please check if the VLAN configuration on the switch matches the VLAN configuration in Proxmox. The switch needs to be configured to allow the VLANs that you're using in Proxmox, and the port connecting to your Proxmox host needs to be configured as a trunk port.
4. Check the VMs' network configuration: Make sure that the VMs are correctly configured to use the right vnet for their network. In particular, ensure that the VMs on your management VLAN are configured to use vmbr0.
5. Test VLAN segregation with tcpdump: You can use tcpdump to check if the VLAN segregation is working correctly. Run tcpdump -i vmbr0 -e -n to show the Ethernet headers and VLAN tags for packets on vmbr0. This can help to confirm whether or not traffic is being tagged correctly.

 

[tsumaru720]

Hi @Hannes Laimer

1. Double-check your network configuration: First, please make sure that your network configuration is correct. In particular, ensure that the VLAN IDs for each of your vnets match their respective VLAN tags. Also, check that you've correctly configured the "vlan-aware" setting on vmbr0.

I've double checked my VLAN IDs on the vnets and they look correct - As far as I understood the "ID" field is just a unique name for the interface, so for i jused named them "vlanxxx" but that could be set to anything really - the important part is the tag field.

Alias is a friendly name and I only have 1 zone which is a VLAN zone configured to use vmbr0
As for the vlan-aware side of things, this is what my vmbr0 looks like - note its actually a bond instead of eth1 (used eth1 for simplicity in explaining, but since im going into way more detail here, might as well elaborate - I dont think it changes much)

iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode active-backup
        bond-primary eno1

iface vmbr0 inet static
        address xxx
        gateway xxx
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

# brctl show
bridge name     bridge id               STP enabled     interfaces
vlan110         8000.00221965528e       no              tap143i0
                                                        veth138i0
                                                        vmbr0.110
vlan120         8000.00221965528e       no              veth274i0
                                                        veth277i0
                                                        vmbr0.120
vlan130         8000.00221965528e       no              vmbr0.130
vlan150         8000.00221965528e       no              tap501i0
                                                        tap9901i1
                                                        vmbr0.150
vlan999         8000.00221965528e       no              tap9901i0
                                                        vmbr0.999
vmbr0           8000.00221965528e       no              bond0
                                                        tap901i0

2. Check for MAC address conflicts: The output from brctl show shows that all your bridges have the same bridge id (8000.00221965528e). This could potentially cause issues if multiple bridges are trying to handle traffic for the same MAC address. Please check to ensure that there are no MAC address conflicts on your network.

I don't believe there are any duplicate mac addresses - I just let Promxox generate them and I dont do anything like setting them manually anywhere. A cursory glance through everything looks fine

3. Check for VLAN configuration on your switch: If you have a managed switch, please check if the VLAN configuration on the switch matches the VLAN configuration in Proxmox. The switch needs to be configured to allow the VLANs that you're using in Proxmox, and the port connecting to your Proxmox host needs to be configured as a trunk port.

I do have a managed switch

interface ethernet 0/0/4
description PVE2
switchport default vlan 100
no switchport hybrid vlan 1
switchport hybrid tagged vlan 101,110,120,130,150,999
switchport hybrid untagged vlan 100
exit

Vlan 100 - untagged, is my management vlan. It's what the hosts and switch/router management interfaces sit on - 101 isnt used, the rest are for VMs.

This worked just fine with my old network config described in my previous post so I dont believe my switch config is at fault.

4. Check the VMs' network configuration: Make sure that the VMs are correctly configured to use the right vnet for their network. In particular, ensure that the VMs on your management VLAN are configured to use vmbr0.

VMs/Containers that are set to use the vnet interfaces seem to be working just fine - I correctly see traffic inside them without a VLAN tag so the tag is being removed before its sent to the VM as I'd expect - I do not make use of the vlan tag field on the actual VM network config
And only see traffic for subnets I expect


The VM I see this problem on is similar, except its KVM, using virtio and its "bridge" is vmbr0


5.Test VLAN segregation with tcpdump: You can use tcpdump to check if the VLAN segregation is working correctly. Run tcpdump -i vmbr0 -e -n to show the Ethernet headers and VLAN tags for packets on vmbr0. This can help to confirm whether or not traffic is being tagged correctly.

Doing this on vmbr0 on the host, I do see all the vlan tags correctly in the output. Hopping into a container or VM that's on a VLAN, I only see traffic I expect as per point 4.

Jumping into the VM that's attached to vmbr0 then, similarly to the host, I see a whole bunch of traffic and all the vlan tags

---

I suspect this whole thing is just a quirk of using the vlan aware option and then attaching a vm directly to the bridge.

From what I understand, if I disable that option, then I'd get a whole bunch of bridges configured such as vmbr0v110 etc which would essentially mimic my old config so I may just look to do that.

From the docs: creating a guest on VLAN 5 for example, would create two interfaces eno1.5 and vmbr0v5, which would remain until a reboot occurs

I was just hoping to make use of the vlan-aware feature, but its not the end of the world if I cant - I'm just trying to rule out an issue with my configuration that'd prevent it from working

 

[spirit]

Note that if if was working fine before using sdn, It's not mandatory to enable vlan-aware on vmbr0 for sdn use. (if not enable, sdn will generate same kind for bridge contruct, tagging the physical eth1.X).

Basically, vlan-aware is making tag at bridge level && non-vlanaware is making tag at physical interface level.

 

[tsumaru720]

Thanks @spirit - i think that'll be my answer...turn off vlan-aware and let it just do the extra interfaces/bridges