How to setup VLAN in Proxmox Cluster for VM communication on different nodes

[leonidas_o]

Hello guys,

I can't get the communication right between VM's on different Nodes (Proxmox Cluster).
I read some posts about VLAN and VXLAN (Proxmox SDN experimental feature).

I decided to try it with VLANs first, as VXLAN is also marked experimental. I'm not even sure, if it's achievable that way.
So I would like that the VM's can "see each other" even though some are on node1 and the others on node2.
I'm using vSwitch (Hetzner), haven't ordered any extra subnets. So just created the vSwitch, which has the VLAN ID 4000. Moved all the physical servers into that one VLAN.

I have the following /etc/network/interfaces file contents on both proxmox nodes. Then on both nodes, inside the VM's Hardware view, set vmbr2 and VLAN Tag 4000.
When now trying to ping the second VM from within the first VM or vice versa, it says Destination Host Unreachable.
Am I missing something here or isn't that even possible that way?

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
  address IPADDRESS
  netmask 255.255.255.192
  gateway NODEGATEWAY
  up route add -net IPADDRESS netmask 255.255.255.192 gw IPADDRESS dev enp0s31f6

iface enp0s31f6 inet6 static
  address 2...
  netmask 64
  gateway f...


auto vmbr2.4000
iface vmbr2.4000 inet static
        address 10.2.3.1/24
        mtu 1400

auto vmbr0
iface vmbr0 inet static


...


auto vmbr2
iface vmbr2 inet static
        #address 10.2.3.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

        post-up   echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.2.3.0/24' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.2.3.0/24' -o enp0s31f6 -j MASQUERADE

 

[leonidas_o]

I was informed that one of my servers (NFS) can't be upgraded to 10Gbit, when using vSwitch, I would have to order a completely new server with 10Gbit uplink etc. So actually 10Gbit is not usable as for now.

So I'm curious, what are actually the benefits (performance, setup etc.) between Hetzner vSwitch and Proxmox SDN - EVPN (BGP EVPN), as this seems to be the best approach from all mentioned variants in the docs (VLAN, QinQ, VXLAN, Simple, EVPN).