[SOLVED] Traffic not forwarded over Vxlan using pre 8 config

[hk135]

Hi All,

Thanks for any help in advance

I am having an odd one whilst trying to upgrade my cluster to proxmox 8, I have Openvswitch configured using vxlan to create virtual networks using a star topology. An example of my config is:

Center Node (192.168.1.40, pve 7.4):
allow-ovs vmbr15
auto vmbr15
iface vmbr15 inet static
address 192.168.15.6/28
gateway 192.168.15.1
ovs_type OVSBridge
post-up ovs-vsctl add-port vmbr15 tun15-int-ovs-rtr -- set interface tun15-int-ovs-rtr type=geneve options:remote_ip=192.168.1.31 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve1 -- set interface tun15-pve1 type=geneve options:remote_ip=192.168.1.42 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve2 -- set interface tun15-pve2 type=geneve options:remote_ip=192.168.1.43 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve-archie1 -- set interface tun15-pve-archie1 type=geneve options:remote_ip=192.168.1.45 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve-archie3 -- set interface tun15-pve-archie3 type=geneve options:remote_ip=192.168.1.41 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve-storage1 -- set interface tun15-pve-storage1 type=vxlan options:remote_ip=192.168.1.44 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve-storage2 -- set interface tun15-pve-storage2 type=geneve options:remote_ip=192.168.1.46 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve-charlie -- set interface tun15-pve-charlie type=geneve options:remote_ip=192.168.1.47 options:key=15
post-up ovs-vsctl add-port vmbr15 tun15-pve-archie2 -- set interface tun15-pve-archie2 type=geneve options:remote_ip=192.168.1.48 options:key=15

Spoke Node (192.168.1.46, pve 7.4):

allow-ovs vmbr15
auto vmbr15
iface vmbr15 inet static
address 192.168.15.4/28
gateway 192.168.15.1
ovs_type OVSBridge
post-up ovs-vsctl set Bridge vmbr15 rstp_enable=true
post-up ovs-vsctl add-port vmbr15 tun15-pve-archive -- set interface tun15-pve-archive type=geneve options:remote_ip=192.168.1.40 options:key=15

Broken Node (192.168.1.44, pve 8.0.2):
allow-ovs vmbr15
auto vmbr15
iface vmbr15 inet static
address 192.168.15.3/28
gateway 192.168.15.1
ovs_type OVSBridge
post-up ovs-vsctl set Bridge vmbr15 rstp_enable=true
post-up ovs-vsctl add-port vmbr15 tun15-pve-archive -- set interface tun15-pve-archive type=vxlan options:remote_ip=192.168.1.40 options:key=15

I switched the broken node from geneve to vxlan to confirm if the tunnelling protocol was at fault but the same issue,I tried upgrading using nosub and same issue. All the 7.4 nodes work just fine but it seems like the 8.0 node won't forward or recieve from a tunnelled port for some reason..
On the broken node I tried adding a veth pair and running tcpdump on the veth and I can see rstp and arp requests from the broken node just fine but nothing from anywhere else. I have confirmed there is something listening on udp port 4789:

udp 0 0 0.0.0.0:4789 0.0.0.0:* -
udp6 0 0 :::4789 :::* -

I have also confirmed the 192.168.1.0/24 interfaces are all running with mtu 9000 so the encapsulated interfaces set to 1500 mtu should be fine.

It occurred to me there maybe something blocking as part of the new SDN feature, but I am unsure how to confirm that.

Any help is appreciated.

Thanks

 

[spirit]

sdn feature don't use ovs, and only writing config in /etc/network/interfaces.d/sdn. (so , it's really not related).

Maybe it's a kernel bug. (maybe try to downgrade to 5.15 kernel) or an ovs bug.

 

[hk135]

So in typical fashion I found a work around for this fairly quickly after. Restarting Openvswitch solves the issue so I had added a cron job to restart the openvswitch-switch 1 min after reboot. Seems to fix it:

@reboot /bin/bash -c "sleep 60; /usr/share/openvswitch/scripts/ovs-systemd-reload"

 

[hk135]

Just to add, this appears to be present in the base Debian Bookworm as well so rather than a Proxmox issue its seems to be a Debian issues. To get DHCP6 to work over a Openvswitch bridge I had to do similar to above, but I ran it in the interfaces file:

# The primary network interface
auto ens18
iface ens18 inet static
address 192.168.1.11/24
netmask 255.255.255.0
gateway 192.168.1.1
mtu 9000
post-up /bin/bash -c "sleep 20; /usr/share/openvswitch/scripts/ovs-systemd-reload"

auto edge6-br
iface edge6-br inet static
address 10.0.0.1/24
iface edge6-br inet6 dhcp