VXLAN encryption

M

mdenadal

Active Member
Mar 23, 2018
29
4
43
54
I manage a 4 nodes Proxmox cluster.
Nodes are located in two different datacenters and connected through public network.
Till SDN there was no L2 shared between nodes private (aka host only) network.
Using SDN (eg. vxlan zones) it's possible to distribute interconnected bridges allowing a bunch of really nice features.

The problem is that vxlan does not currently implement any type of encryption.
Obviously using public network connections without encryption is not viable.
Spirit said that MACsec is on the way (probably later this year) but at the moment I think the simplest way is to use ipsec for vxlan connections.

Configuration is really simple, let's say you have a 3 nodes cluster,
first of all you need to install strongswan on each Proxmox box:
Code: 
root@node1 /etc # apt install strongswan

even if it is not necessary I suggest to insert nodes address in hosts file.
Obviously this is only an example:
Code: 
root@node1 / # cat /etc/hosts
127.0.0.1        locahost.localdomain localhost
173.x.y.z        node1
167.x.y.z        node2
80.x.y.z         node3

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Then you have to configure your ipsec tunnels.
Edit /etc/ipsec.conf (this is the fast way, using /etc/ipsec.d/ files could be better):
Code: 
root@node1 / # cat /etc/ipsec.conf
conn common
    authby=secret
    auto=start              
    ike=aes256-sha1-modp1024!  # in my tests these seem to be the fastest (but reasonably secure)ciphers on reasonably modern hardware
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox firewall rules
    left=node1
    leftsubnet=node1[udp/4789] # encrypt only vxlan traffic

conn node1-to-node2            # the idea is to full mesh your nodes
    also=common
    right=node2
    rightsubnet=node2[udp/4789]

conn node1-to-node3
    also=common
    right=node3
    rightsubnet=node3[udp/4789]

add preshared keys editing /etc/ipsec.secrets:
Code: 
root@node1 / # cat /etc/ipsec.secrets
node1 node2 : PSK "superstrongpassword"
node1 node3 : PSK "superstrongpassword"

obviously you have to properly edit ipsec.conf and ipsec.secrets on each node.

Finally you can start strongswan on each node:
Code: 
root@node1 / # ipsec restart

and check if everything is running fine:
Code: 
root@node1 / # ipsec statusall

to start encryption at boot time:
Code: 
root@node1 / # systemctl enable ipsec

Just my two cents.
Cheers.
Massimo
 
Last edited:
  • Like
Reactions: Wichets and bobmc
spirit said:
do you have reduce the mtu to handle the ipsec encryption ?
Click to expand...

After some tests I saw no performance difference between 1450 and lower MTU.
So I stick to 1450 but to be honest I haven't verified if fragmentation occours.
From an operational point of view, everything seems ok with MTU 1450.
 
Hi,

I have looked at macsec, but it'll be too difficult to implement, because I need to add mac address of all vms dynamicly on the macsec interfaces. (as it's done inside the vxlan).

So I think I'm going to implemented ipsec (in transport mode as we already have vxlan tunnel) , don't known yet if I'll use strongswan with ike for key rotation, or directly with iproute2 commands with static keys.

I'll look also for wireguard in the future.

I don't have tested your strongswan config yet, does it working fine ?
 
spirit said:
I'll look also for wireguard in the future.

I don't have tested your strongswan config yet, does it working fine ?
Click to expand...

Hi Spirit,
in my humble opinion Wireguard is definetely the way to go.
Way easier to setup and very resilient connections even on poor networks.
In the meantime my dirty ipsec setup is up and running since july.
 
mdenadal said:
Hi Spirit,
in my humble opinion Wireguard is definetely the way to go.
Way easier to setup and very resilient connections even on poor networks.
In the meantime my dirty ipsec setup is up and running since july.
Click to expand...
I'm waiting official wireguard support in a newer kernel.
thanks for the hints for ipsec.
 
here a simpler config for any number of hosts, for udp 4789 only.
for traffic between any host on this port.

with this config, no need to do change when you are adding new nodes.


Code: 
conn %default
    ike=aes256-sha1-modp1024!  # in my tests these seem to be the fastest (but reasonably secure)ciphers on reasonably modern hardware
    esp=aes256-sha1!
    leftfirewall=yes           # this is necessary when using Proxmox firewall rules

conn output
    rightsubnet=%dynamic[udp/4789]
    right=%any
    type=transport
    authby=psk
    auto=route


conn input
    leftsubnet=%dynamic[udp/4789]
    type=transport
    authby=psk
    auto=route

with ipsec.secrets (same psk key for everybody)

Code: 
: PSK 0sv+NkxY9LLZvwj4qCC2o/gGrWDF2d21jL
 
about mtu,
using transport mode instead tunnel, used 20bytes less.

but still,

ipsec in transport mode + aes + sha1 -> around 60bytes in ipv4 && 80bytes with ipv6

I have found multiple article, with not exatcly the same maths

https://www.fortinetguru.com/2019/06/ipsec-vpn-concepts-3/5/
https://packetpushers.net/ipsec-bandwidth-overhead-using-aes/
https://hamwan.org/Standards/Network Engineering/IPsec.html

with the 50bytes for vxlan, with a mtu 1500 by default, it should be reduce around 1370-1390.
 
Hi,
for information in a 4 nodes cluster, in order to keep the connections up, I had to add the following parameters in /etc/ipsec.conf:

Code: 
config setup
    uniqueids=no
Without this parameters I had many disconnection with this message:

Code: 
[IKE] schedule delete of duplicate IKE_SA for peer xx.xx.xx.xx  due to uniqueness policy and suspected reauthentication
 
Last edited:
sharky69 said:
Hi,
for information in a 4 nodes cluster, in order to keep the connections up, I had to add the following parameters in /etc/ipsec.conf:


Code: 
config setup
    uniqueids=no
Without this parameters I had many disconnection with this message:

Code: 
[IKE] schedule delete of duplicate IKE_SA for peer xx.xx.xx.xx  due to uniqueness policy and suspected reauthentication
Click to expand...
Thanks for the info !
 
Anyone run into these errors? I am not finding much on how to resolve these.

root@pve6a:~# systemctl status strongswan.service
× strongswan.service - strongSwan IPsec IKEv1/IKEv2 daemon using swanctl
Loaded: loaded (/usr/lib/systemd/system/strongswan.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Tue 2025-10-21 17:25:25 EDT; 5s ago
Invocation: 752e6a26502b4d1abd57c8da1e6c2063
Process: 1685312 ExecStart=/usr/sbin/charon-systemd (code=exited, status=0/SUCCESS)
Process: 1685340 ExecStartPost=/usr/sbin/swanctl --load-all --noprompt (code=exited, status=13)
Main PID: 1685312 (code=exited, status=0/SUCCESS)
Status: "charon-systemd running, strongSwan 6.0.1, Linux 6.14.11-4-pve, x86_64"
Mem peak: 5.2M
CPU: 40ms

Oct 21 17:25:25 pve6a swanctl[1685340]: --raw (-r) dump raw response message
Oct 21 17:25:25 pve6a swanctl[1685340]: --pretty (-P) dump raw response message in pretty print
Oct 21 17:25:25 pve6a swanctl[1685340]: --file (-f) custom path to swanctl.conf
Oct 21 17:25:25 pve6a swanctl[1685340]: --debug (-v) set debug level, default: 1
Oct 21 17:25:25 pve6a swanctl[1685340]: --options (-+) read command line options from file
Oct 21 17:25:25 pve6a swanctl[1685340]: --uri (-u) service URI to connect to
Oct 21 17:25:25 pve6a systemd[1]: strongswan.service: Control process exited, code=exited, status=13/n/a
Oct 21 17:25:25 pve6a charon-systemd[1685312]: SIGTERM received, shutting down
Oct 21 17:25:25 pve6a systemd[1]: strongswan.service: Failed with result 'exit-code'.
Oct 21 17:25:25 pve6a systemd[1]: Failed to start strongswan.service - strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.

root@pve6a:~# journalctl -xeu strongswan.service
Oct 21 17:25:25 pve6a swanctl[1685340]: strongSwan 6.0.1 swanctl
Oct 21 17:25:25 pve6a swanctl[1685340]: usage:
Oct 21 17:25:25 pve6a swanctl[1685340]: swanctl --load-all [--raw|--pretty] [--clear] [--noprompt]
Oct 21 17:25:25 pve6a swanctl[1685340]: --help (-h) show usage information
Oct 21 17:25:25 pve6a swanctl[1685340]: --clear (-c) clear previously loaded credentials
Oct 21 17:25:25 pve6a swanctl[1685340]: --noprompt (-n) do not prompt for passwords
Oct 21 17:25:25 pve6a swanctl[1685340]: --raw (-r) dump raw response message
Oct 21 17:25:25 pve6a swanctl[1685340]: --pretty (-P) dump raw response message in pretty print
Oct 21 17:25:25 pve6a swanctl[1685340]: --file (-f) custom path to swanctl.conf
Oct 21 17:25:25 pve6a swanctl[1685340]: --debug (-v) set debug level, default: 1
Oct 21 17:25:25 pve6a swanctl[1685340]: --options (-+) read command line options from file
Oct 21 17:25:25 pve6a swanctl[1685340]: --uri (-u) service URI to connect to
Oct 21 17:25:25 pve6a systemd[1]: strongswan.service: Control process exited, code=exited, status=13/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStartPost= process belonging to unit strongswan.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 13.
Oct 21 17:25:25 pve6a charon-systemd[1685312]: SIGTERM received, shutting down
Oct 21 17:25:25 pve6a systemd[1]: strongswan.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit strongswan.service has entered the 'failed' state with result 'exit-code'.
Oct 21 17:25:25 pve6a systemd[1]: Failed to start strongswan.service - strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.
░░ Subject: A start job for unit strongswan.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit strongswan.service has finished with a failure.
░░
░░ The job identifier is 11019 and the job result is failed.
lines 346-380/380 (END)


Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'test-vectors': failed to load - test_vectors_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'ldap': failed to load - ldap_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'pkcs11': failed to load - pkcs11_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'aes': failed to load - aes_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'rc2': failed to load - rc2_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'sha2': failed to load - sha2_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'sha1': failed to load - sha1_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'md5': failed to load - md5_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'mgf1': failed to load - mgf1_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'rdrand': failed to load - rdrand_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'pkcs12': failed to load - pkcs12_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'pgp': failed to load - pgp_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'gcrypt': failed to load - gcrypt_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'af-alg': failed to load - af_alg_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'fips-prf': failed to load - fips_prf_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'gmp': failed to load - gmp_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'curve25519': failed to load - curve25519_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'chapoly': failed to load - chapoly_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'xcbc': failed to load - xcbc_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'cmac': failed to load - cmac_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'hmac': failed to load - hmac_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'kdf': failed to load - kdf_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'ctr': failed to load - ctr_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'ccm': failed to load - ccm_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: plugin 'curl': failed to load - curl_plugin_create not found and no plugin file available
Oct 21 17:25:25 pve6a swanctl[1685340]: opening socket 'unix:///var/run/charon.vici' failed: Permission denied
Oct 21 17:25:25 pve6a swanctl[1685340]: Error: connecting to 'default' URI failed: Permission denied
Oct 21 17:25:25 pve6a swanctl[1685340]: strongSwan 6.0.1 swanctl
Oct 21 17:25:25 pve6a swanctl[1685340]: usage:
Oct 21 17:25:25 pve6a swanctl[1685340]: swanctl --load-all [--raw|--pretty] [--clear] [--noprompt]
Oct 21 17:25:25 pve6a swanctl[1685340]: --help (-h) show usage information
Oct 21 17:25:25 pve6a swanctl[1685340]: --clear (-c) clear previously loaded credentials
Oct 21 17:25:25 pve6a swanctl[1685340]: --noprompt (-n) do not prompt for passwords
Oct 21 17:25:25 pve6a swanctl[1685340]: --raw (-r) dump raw response message
Oct 21 17:25:25 pve6a swanctl[1685340]: --pretty (-P) dump raw response message in pretty print
Oct 21 17:25:25 pve6a swanctl[1685340]: --file (-f) custom path to swanctl.conf
Oct 21 17:25:25 pve6a swanctl[1685340]: --debug (-v) set debug level, default: 1
Oct 21 17:25:25 pve6a swanctl[1685340]: --options (-+) read command line options from file
Oct 21 17:25:25 pve6a swanctl[1685340]: --uri (-u) service URI to connect to
Oct 21 17:25:25 pve6a systemd[1]: strongswan.service: Control process exited, code=exited, status=13/n/a
Oct 21 17:25:25 pve6a charon-systemd[1685312]: SIGTERM received, shutting down
Oct 21 17:25:25 pve6a systemd[1]: strongswan.service: Failed with result 'exit-code'.
Oct 21 17:25:25 pve6a systemd[1]: Failed to start strongswan.service - strongSwan IPsec IKEv1/IKEv2 daemon using swanctl.
 
You must log in or register to reply here.
Top Bottom