[SOLVED] NGINX with kTLS on unprivileged LXC on Proxmox 7.3

[jsabater]

Good day everyone!

I have a 5-node cluster on PVE 7.3-6 with a couple hundred unprivileged LXC, all using the Debian 11 Bullseye template.

I was looking at how Netflix can serve 800Gb/s of TLS encrypted video content from a single server, and a large part of it appears to be kTLS.

I also saw a blog post from Nginx about improving Nginx performance with Kernel TLS and SSL_sendfile() using recent versions of Linux. As can be read in this article, the original Debian 11 could not be used because its kernel was built with the CONFIG_TLS=n option, but it was later added in 5.15.1. I have checked the nodes of my cluster and they all have the tls loaded (being used by bonding, incidentally), and it appears as such when checking from inside the LXC:

# lsmod | grep tls
tls                   110592  1 bonding

So I thought let's check it out, let's enable kTLS on the Nginx servers. I run version 1.18 at the moment via Debian repo but I created a new LXC (Debian 11) and installed the latest mainline version of NGINX via its repos at nginx.org (1.23.3 at the time of writing). Then I configured NGINX to use it:

http {
  [..]
  sendfile on;
  [..]
}

server {
  listen 443 ssl http2;
  server_name domain.com;
  ssl_certificate /etc/ssl/certs/domain.com.crt;
  ssl_certificate_key /etc/ssl/certs/domain.com.key;
  ssl_protocols TLSv1.3;
  ssl_conf_command Options KTLS;
  [..]
}

But when I try to start the server I get this error:

[emerg] 364#364: SSL_CONF_cmd("Options", "KTLS") failed (SSL: error:1414E180:SSL routines:SSL_CONF_cmd:bad value:cmd=Options, value=KTLS)

I am posting here because of two reasons:

1. I think that the fact that the LXC is unprivileged should not prevent this feature from being used, but I'm not sure. Could anyone confirm?b
2. I have not been able to find information on this error, and I don't know whether to search on the LXC side or the NGINX side of it. The only conclusion I have arrived at is that Debian 11 (and, therefore, Proxmox) ships with OpenSSL 1.1.1n, therefore no way this is going to work until Proxmox is based on Debian 12 Bookworm, which will include OpenSSL 3.0.8.

Any feedback would be very much appreciated.

 

[cheiss]

I have not been able to find information on this error, and I don't know whether to search on the LXC side or the NGINX side of it. The only conclusion I have arrived at is that Debian 11 (and, therefore, Proxmox) ships with OpenSSL 1.1.1n, therefore no way this is going to work until Proxmox is based on Debian 12 Bookworm, which will include OpenSSL 3.0.8.

Since the host kernel supports kTLS already, the Proxmox VE version (and/or the Debian release it is built on) does not matter.

As you correctly identified, Debian 11 ships with an older OpenSSL version, which does not support kTLS - only OpenSSL >3, which only first ships with the upcoming Debian Bookworm release. Meaning, you only need OpenSSL 3.x inside your LXC container and do not need to wait for the next Proxmox VE release.

One thing you can do is to compile your own NGINX + OpenSSL as described in the linked article under "Building NGINX with kTLS on Linux Distributions".
Be sure to create a backup (or snapshot) of your LXC container before doing this!
Just to be on the safe side, in case something breaks.

Before doing that, install all needed dependencies in your container:

apt install gcc make libpcre3-dev zlib1g-dev

After that, follow the instruction in the article. If you have make install'd the newly built NGINX, create the following systemd service file at /etc/systemd/system/nginx-ktls.service:

[Unit]
Description=nginx - high performance web server
Documentation=https://nginx.org/en/docs/
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/var/run/nginx.pid
ExecStart=/usr/local/sbin/nginx -c /etc/nginx/nginx.conf
ExecReload=/bin/sh -c "/bin/kill -s HUP $(/bin/cat /var/run/nginx.pid)"
ExecStop=/bin/sh -c "/bin/kill -s TERM $(/bin/cat /var/run/nginx.pid)"

[Install]
WantedBy=multi-user.target

Now, run:

systemctl disable --now nginx # Disable the "official" nginx
systemctl daemon-reload
systemctl enable --now nginx-ktls

.. and nginx should now start with kTLS enabled.

As the newly compiled NGINX lives in /usr/local, it can safely live side-by-side with mainline NGINX as fallback (as long as only ever one of them is enabled).

 

[jsabater]

Thank you very much for you insightful reply, @cheiss. Yesterday night I managed to get it working, but I had not had the chance to post a reply here.

So, what I did in the end was to install Ubuntu 22.04 (Proxmox template) on a new (unprivileged, as always) LXC, as it ships with OpenSSL 3.0.2. Then I modified my Ansible playbook to install the mainline version of Nginx (1.23.3 at this moment) from the repository at nginx.org. And, well, it worked flawlessly! It actually took much more time for me to adapt my Ansible playbooks than anything else, heh!

Of course, as soon as a LXC template for Debian 12 is available, I'll migrate the NGINX LXC back to it, heh

So, all in all, as you pointed out and for future reference:

1. You don't need a privileged LXC.
2. kTLS is already loaded on the hosts/nodes in Proxmox 7.3.
3. All you need is OpenSSL > 3.0, either by compiling from sources or by using a LXC template with a distro that ships it (e.g. Ubuntu 22.04, Debian 12, etc).

Cheers!