[SOLVED] SDN: VXLAN with BGP EVPN. Cannot add VLAN tag to guest interface

[DEHAAS]

Hi,

I am testing VXLAN with BGP EVPN with Proxmox 7.1 SDN. BGP and EVPN seem to be working fine, and I can route traffic on VM interfaces that do not specify a VLAN tag. However, when I try to set a VLAN tag on a guest VM NIC which is using a VXLAN VNET, I get the following error:

no physical interface on bridge 'test'
kvm: -netdev type=tap,id=net1,ifname=tap101i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on: network script /var/lib/qemu-server/pve-bridge failed with status 6400
TASK ERROR: start failed: QEMU exited with code 1

'test' is the name of the VNET. I have tried with and without setting the VLAN Aware flag on the VNET, but the same result. Any ideas?

 

[spirit]

This is expected. The vlan option in vm nic gui can't be used with vnet && evpn. (and vlan-aware bridge are not yet compatible with evpn/frr)
it should be greyout in the gui,but it's not yet implemented.

 

[DEHAAS]

Hi Spirit,

Thank you very much for your answer! Okay, so that means we cannot expose VLANs inside of VXLANs in Proxmox networking, and we should not be setting the vlan-aware flag? In that case, would it still be fine to use VLAN tags inside of the VMs, or are we not able to use VLANs inside of VXLAN at all? If it should work with splitting on VLANs inside of VMs, I assume we would need to modify the MTU on Proxmox bridges to accommodate?

 

[DEHAAS]

Otherwise, is there a way to bridge multiple VXLAN interfaces in Proxmox to a single interface with VLAN tags for each of the de-encapsulated VXLANs?

 

[spirit]

Otherwise, is there a way to bridge multiple VXLAN interfaces in Proxmox to a single interface with VLAN tags for each of the de-encapsulated VXLANs?

do you need to bridge a vlan to a vxlan ?
If yes,it's currently not implemented.
the vnet itself, is not bridged to the physical network.
It's possible to do it with 1 host with some tricks

for example, with adding in:

/etc/network/interfaces

auto yourvnetvxlan100
iface yourvnetvxlan100
    bridge-ports ethX.100

auto yourvnetvxlan200
iface yourvnetvxlan200
    bridge-ports ethX.200

(This will be merged will the generated vnet config in /ec/nework/interfaces.d/sdn).

But with 2 hosts, you'll have network loop. (This should need to implement some kind of mlag between 2 proxmox node, but I don't known any opensource implementation on linux).


Personnaly, at work, I'm doing it with a pair of arista switch in mlag (with evpn).

 

[DEHAAS]

Again, thank you very much for your help.

This is not exactly what I am trying to achieve. I am trying to build infrastructure to support a multi-tenant environment. Currently, we have a couple of hypervisors (not Proxmox) connected to two Dell OS10 switches with MLAG. On these hypervisors, we have multiple distinct tenants, each with multiple VLANs. These hosts are using plan VLANs, not VXLANs. The Dell switches are now encapsulating each of these VLANs into VXLANs and announcing them to the Proxmox nodes. Unfortunately, it seems that the Dell switches cannot be configured to keep the VLAN tag inside the VXLAN. However, I would still like to be able to bundle VLANs per tenant into a single virtual trunk interface on the Proxmox nodes. This would allow a VM in Proxmox to access all VLANs for a specific tenant with a single virtual NIC. Do you see any way to achieve this?

 

[spirit]

do you need to extend a specific subnet in a vlan<->vxlan layer2 ? (for ex: vlan20: 192.168.1.0/24 vxlan10: 192.168.1.0/24)

or do you need to route traffic between differents subnets ? (for ex: vlan20: 192.168.1.0/24 vxlan10: 192.168.2.0/24).

(The second option can be achieved with the exit-nodes)

. The Dell switches are now encapsulating each of these VLANs into VXLANs and announcing them to the Proxmox nodes. Unfortunately, it seems that the Dell switches cannot be configured to keep the VLAN tag inside the VXLAN

Are you doing evpn with your dell switch too ?

Generally, the physical switchs os are mapping vlan to vxlan. (vlan10<-> vxlan10, vlan11 <->vxlan11,..). Some implementations are able to map multipe vxlan in 1 tunnel. It's not implemented yet in proxmox because it's need a recent kernel.
https://docs.nvidia.com/networking-...ualization/VXLAN-Devices/#single-vxlan-device

 

[spirit]

Hi,
I'm currently looking the dell documentation.
So if I understand your request, you are doing evpn from your dell switchs, and vlan are mapped to vxlan by dell switch to evpn. (mapped/translated, not "encapsulated", this is important, because vlan tag is not inside a vxlan tunnel. the vlan tag is replaced by a vxlan).

So, no need to use vlan-aware bridge on proxmox side.

The implementation on proxmox side is symetric. (if you need to do routing a with anycast gateway)
https://www.dell.com/support/manual...f2c46a-a9cc-43ce-bab7-f68e123d9eeb&lang=en-us


I don't known if os10 is doing multiple vxlan inside 1 tunnel, or 1tunnel=1vxlan (the implementation on proxmox side),
but I remember to have helped a user of the forum with a similar dell switch and it's was compatible
https://forum.proxmox.com/threads/sdn-bgp-evpn-not-working.107334/#post-462018

So I think it should work.

Maybe can you send your os10 config ?

 

[DEHAAS]

Thank you so much for your help @spirit. It turns out that I cannot get around the current limitations on Dell OS10, but it sounds like they will be lifted in the next minor firmware release of OS10 (10.5.4.0). Once again. Your help here is very highly appreciated!

 

[spirit]

Thank you so much for your help @spirit. It turns out that I cannot get around the current limitations on Dell OS10, but it sounds like they will be lifted in the next minor firmware release of OS10 (10.5.4.0). Once again. Your help here is very highly appreciated!

ok no problem.

if you are able to get it working after minor firmware, try to share your config on the forum, it could help other proxmox users too