Unexpected vxlan behaviour with vlan

[jmellipse]

Hi,

I have 3 Proxmox 7.1-12 nodes in HA.
They are connected with serveral 10GbE cards to 100GbE Cumulus Linux switches.
The Cumulus switches use vxlan over an frr/ospf underlay.

Testing is on a single Proxmox host currently. I want to include the ovs VM bridge in the ospf underlay in order the have vxlan tunnels from the physical Cumulus switches up to the VM bridge on the Proxmox host.

This works fine, until I configure vlan inside the vxlan. It seems that vlan tags are droped on the Proxmox side no matter what I configure.

I sent traffic from a physical node on vlan 100 to the Proxmox VM.
As long as the VM on Proxmox has no vlan configured on the bridge or inside the VM, I get traffic just fine. But I expected it on vlan 100, either with a tagged bridge or configured inside the VM.

What I'm I missing? Is this behaviour a misconfiguration or maybe due to OVS behaviour?

##########

Configuration:

Configured an ovs bridge on Proxmox as:

auto vmbr1 iface vmbr1 inet static address 10.255.21.47/24 ovs_type OVSBridge ovs_ports enp130s0f1 mtu 9200 post-up ifconfig enp130s0f1 mtu 9200 && ifconfig vmbr1 mtu 9200

I configured frr/zebra on the Proxmox host to advertise the routes, this work perfect. I can ping and traceroute the complete underlay. I can confirm this on the Proxmox host with vtysh.

To test vxlan I installed a physical node with ubuntu.

Ubuntu Physical node vlan part:

vlans: ens1.100: addresses: - 10.11.12.10/24 id: 100 link: ens1

Cumulus switch:

auto vni313 iface vni313 vxlan-id 313 vxlan-local-tunnelip 10.255.255.3 vxlan-remoteip 10.255.255.47 mtu 9200

and

auto br313 iface br313 bridge-ports vni313 swp27s0.100 bridge-stp off

On the Proxmox host the SDn bridge:

auto br313 iface br313 bridge_ports vxlan_br313 bridge_stp off bridge_fd 0 mtu 1500 auto vxlan_br313 iface vxlan_br313 vxlan-id 313 vxlan_remoteip 10.255.255.3 mtu 9200

 

[spirit]

1) about vlan-> vxlan
on in your cumulus config:
bridge-ports vni313 swp27s0.100

That's mean than when traffic is coming from vlan 100 (from swp27s0), then vlan tag is removed. (as it's not a vlan aware bridge, the vlan tag is dropped). Then the packet is is forwarded to vni313.
This is on cumulus side, not proxmox side. If you need to keep vlans inside the vxlan tunnel, you need to use vlan-aware bridge on both side.

But, I'm not sure that cumulus support it, if I remember I have added a patch to ifupdown2 to support vlan inside vxlan tunnel. The cumulus way is more 1vlan = 1 vxlan.

2) not related, but what is the mtu inside your vm ?

vxlan add 50bytes when traffic is going out the vxlan interface. (so here,through the br313 bridge).

if you have 1500 bytes mtu in your vm, you'll have problem because of mtu 1500 on br313. (on the proxmox host)

 

[jmellipse]

Thank you for your explanation, that makes perfect sense.

One thing I forgot to mention is that I'm not much of a network guy. I was indeed under the impression that the vlan taged was passed in Cumulus.

Cumulus does have a vlan aware bridge, but you can only have one per switch.