VXLAN encryption

[mdenadal]

I manage a 4 nodes Proxmox cluster.
Nodes are located in two different datacenters and connected through public network.
Till SDN there was no L2 shared between nodes private (aka host only) network.
Using SDN (eg. vxlan zones) it's possible to distribute interconnected bridges allowing a bunch of really nice features.

The problem is that vxlan does not currently implement any type of encryption.
Obviously using public network connections without encryption is not viable.
Spirit said that MACsec is on the way (probably later this year) but at the moment I think the simplest way is to use ipsec for vxlan connections.

Configuration is really simple, let's say you have a 3 nodes cluster,
first of all you need to install strongswan on each Proxmox box:

root@node1 /etc # apt install strongswan

even if it is not necessary I suggest to insert nodes address in hosts file.
Obviously this is only an example:

root@node1 / # cat /etc/hosts
127.0.0.1        locahost.localdomain localhost
173.x.y.z        node1
167.x.y.z        node2
80.x.y.z         node3

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Then you have to configure your ipsec tunnels.
Edit /etc/ipsec.conf (this is the fast way, using /etc/ipsec.d/ files could be better):

root@node1 / # cat /etc/ipsec.conf
conn common
    authby=secret
    auto=start              
    ike=aes256-sha1-modp1024!  # in my tests these seem to be the fastest (but reasonably secure)ciphers on reasonably modern hardware
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox firewall rules
    left=node1
    leftsubnet=node1[udp/4789] # encrypt only vxlan traffic

conn node1-to-node2            # the idea is to full mesh your nodes
    also=common
    right=node2
    rightsubnet=node2[udp/4789]

conn node1-to-node3
    also=common
    right=node3
    rightsubnet=node3[udp/4789]

add preshared keys editing /etc/ipsec.secrets:

root@node1 / # cat /etc/ipsec.secrets
node1 node2 : PSK "superstrongpassword"
node1 node3 : PSK "superstrongpassword"

obviously you have to properly edit ipsec.conf and ipsec.secrets on each node.

Finally you can start strongswan on each node:

root@node1 / # ipsec restart

and check if everything is running fine:

root@node1 / # ipsec statusall

to start encryption at boot time:

root@node1 / # systemctl enable ipsec

Just my two cents.
Cheers.
Massimo

 

[spirit]

Thanks for sharing the config !

do you have reduce the mtu to handle the ipsec encryption ?

 

[mdenadal]

do you have reduce the mtu to handle the ipsec encryption ?

After some tests I saw no performance difference between 1450 and lower MTU.
So I stick to 1450 but to be honest I haven't verified if fragmentation occours.
From an operational point of view, everything seems ok with MTU 1450.

 

[spirit]

Hi,

I have looked at macsec, but it'll be too difficult to implement, because I need to add mac address of all vms dynamicly on the macsec interfaces. (as it's done inside the vxlan).

So I think I'm going to implemented ipsec (in transport mode as we already have vxlan tunnel) , don't known yet if I'll use strongswan with ike for key rotation, or directly with iproute2 commands with static keys.

I'll look also for wireguard in the future.

I don't have tested your strongswan config yet, does it working fine ?

 

[mdenadal]

I'll look also for wireguard in the future.

I don't have tested your strongswan config yet, does it working fine ?

Hi Spirit,
in my humble opinion Wireguard is definetely the way to go.
Way easier to setup and very resilient connections even on poor networks.
In the meantime my dirty ipsec setup is up and running since july.

 

[spirit]

Hi Spirit,
in my humble opinion Wireguard is definetely the way to go.
Way easier to setup and very resilient connections even on poor networks.
In the meantime my dirty ipsec setup is up and running since july.

I'm waiting official wireguard support in a newer kernel.
thanks for the hints for ipsec.

 

[spirit]

here a simpler config for any number of hosts, for udp 4789 only.
for traffic between any host on this port.

with this config, no need to do change when you are adding new nodes.

conn %default
    ike=aes256-sha1-modp1024!  # in my tests these seem to be the fastest (but reasonably secure)ciphers on reasonably modern hardware
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox firewall rules

conn output
    rightsubnet=%dynamic[udp/4789]
    right=%any
    type=transport
    authby=psk
    auto=route


conn input
    leftsubnet=%dynamic[udp/4789]
    type=transport
    authby=psk
    auto=route

with ipsec.secrets (same psk key for everybody)

: PSK 0sv+NkxY9LLZvwj4qCC2o/gGrWDF2d21jL

 

[spirit]

about mtu,
using transport mode instead tunnel, used 20bytes less.

but still,

ipsec in transport mode + aes + sha1 -> around 60bytes in ipv4 && 80bytes with ipv6

I have found multiple article, with not exatcly the same maths

https://www.fortinetguru.com/2019/06/ipsec-vpn-concepts-3/5/
https://packetpushers.net/ipsec-bandwidth-overhead-using-aes/
https://hamwan.org/Standards/Network Engineering/IPsec.html

with the 50bytes for vxlan, with a mtu 1500 by default, it should be reduce around 1370-1390.

 

[sharky69]

Hi,
for information in a 4 nodes cluster, in order to keep the connections up, I had to add the following parameters in /etc/ipsec.conf:

config setup
    uniqueids=no

Without this parameters I had many disconnection with this message:

[IKE] schedule delete of duplicate IKE_SA for peer xx.xx.xx.xx  due to uniqueness policy and suspected reauthentication

 

[spirit]

Hi,
for information in a 4 nodes cluster, in order to keep the connections up, I had to add the following parameters in /etc/ipsec.conf:

config setup
    uniqueids=no

Without this parameters I had many disconnection with this message:

[IKE] schedule delete of duplicate IKE_SA for peer xx.xx.xx.xx  due to uniqueness policy and suspected reauthentication

Thanks for the info !