[SOLVED] VxLan is unstable

[Javier Vilarroig]

Hi.

We have a 5 nodes Proxmox Cluster, but will be reduced t three en the next months.

I have setup an internal network with SDN/VxLAN for the first time.

Following the documentation advise, I'm using IPSEC for improved security.

In this internal network I have deployed dnsmasq for DHCP and DNS service.

The SDN works well. Any LXC in the defined LAN can see the others in a transparent way.

But, every 10m or so the VxLAN connection goes down. the LXC's can reach the ones on the local node but are unable to reach the ones in the other nodes. After two or three minutes, connection comes back.

I have been trying to diagnose and fix the issue for more than a month already, but to no avail.

Any idea of what can I do to fix this issue?

Thanks in advance!!

 

[sw-omit]

Haven't use VxLAN's (didn't have to since all my servers are just in the same, real, lan), but maybe a few general things to check (you might have already done them, but didn't mention it, so just confirming):

• Have you verified that the IPSec VPN itself remains both up and stays stable, maybe try pinging from the hosts to the other hosts over the VPN? And what device is making the IPSec tunnel itself (Proxmox host, VM/LXC on Proxmox, off-host router/device)?
• If you run "full" VM's, do they stop working too?
• Anything in logs at time of drop or re-connect?

 

[spirit]

maybe can you try without ipsec to compare ?

vxlan use udp, it's stateless without any connection.

 

[Javier Vilarroig]

Thanks for the answers!

Finally, I found that the issue was in IPSec.

I'm not an IPSec expert so I blindly copied the configuration recommended by the documentation. But this configuration was affected by by too fast renegotiation, making the network unreliable.

After some trial and error I found a configuration that has ended with the renegotiation issues.

config setup
    uniqueids=no


conn %default
    ike=aes256-sha1-modp1024!  # the fastest, but reasonably secure cipher on modern HW
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox VE firewall rules
    lifebytes=4294967296000
    ikelifetime=28800s
    keyexchange=ikev2
    keylife=3600s
    keyingtries=%forever
    mobike=no

conn output

    rightsubnet=%dynamic[udp/4789]
    right=%any
    type=transport
    authby=psk
    auto=route

conn input
    leftsubnet=%dynamic[udp/4789]
    type=transport
    authby=psk
    auto=route