One LetsEncrypt-certificate for all Clustermember and access behind an haproxy

udo

Distinguished Member
Apr 22, 2009
5,977
201
163
Ahrensburg; Germany
Hi,
I have just install on my home-cluster letsencrypt with the same certificate for all clustermember.
Pro:
- I need recreate one certificate only. (not an big plus)
- I can access the cluster from outside with one address through haproxy (Port 8006) and have an valid certificate equal on which node I logged in. (an real benefit!)

The cert combine all nodename plus the "default-landing-name" for extern:
Code:
Certificate Subject Alt Name:
DNS Name: pve-a.example.com
DNS Name: pve-b.example.com
DNS Name: pve-c.example.com
DNS Name: pve.example.com
If someone interesting in an tutorial I can posted it here.

Udo
 
> If someone interesting in an tutorial I can posted it here.

Yes, its's very cool!
 
Partly covered from german howto https://www.deimeke.net/dirk/blog/index.php?/plugin/tag/letsencrypt

The installation is needed on one cluster-node only - in this example the nodes are pve-a, pve-b and pve-c.example.com .
The first nodes is reachable as 192.168.200.11

The DNS/Firewall must be configured, that pve.example.com, pve-a.example.com, pve-b.exapmle.com and pve-c.example.com are directed to the proxy-server with haproxy (cnames for dns).

Important 1! If you allready used letsencrypt with the "default" method it's very important to use the account.key from /etc/pve/.le/
Otherwise you are not able to get certificates with the same name as allready used!

Important 2!
run an backup of /etc/pve, like:
Code:
tar cvf /root/etc_pve.tar /etc/pve/

Preparations
Code:
# not nesseary if /etc/pve/.le/account.key excist!
mkdir /etc/pve/.le/
openssl genrsa 4096 > /etc/pve/.le/letsencrypt-pve.key
If you used letsencryt before, disable/delete the cronjob (crontab -e) about /root/.acme.sh !
To be sure, move the dir:
Code:
mv /root/.acme.sh /root/.acme.sh.old
Install needed tools:
Code:
apt install libssl-dev gcc unzip make
get AcmeFetch and install
Code:
# download the latest version https://github.com/oetiker/AcmeFetch/releases/latest

wget https://github.com/oetiker/AcmeFetch/releases/download/v0.5.0/acmefetch-0.5.0.tar.gz
tar xvzf acmefetch-0.5.0.tar.gz
cd acmefetch-0.5.0/
./configure --prefix=/opt/acmefetch
/usr/bin/make install
for security remove gcc+make again
Code:
apt remove gcc make
steps for temporary webserver
Code:
mkdir -p /opt/acmefetch/www/.well-known/acme-challenge

cat > /opt/acmefetch/www/index.html << EOF
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type">
<title>temporary website</title>
<meta content="Author" name="pve"></head>
<body>Hello.<br>
The python httpd server is working.<br>
and waiting for letsencrypt<br>
</body></html>
EOF
create script to create/renew cert:
Code:
cat > /usr/local/sbin/le_cert_renew.sh << EOF
#!/bin/bash
#
# le_cert_renew.sh renew certificates if nessesary
# (ul) 20161029 first version

# start simple http-server
cd /opt/acmefetch/www
python3 -m http.server 8080&
http_pid=\$(ps aux | grep "python3 -m http.server 8080" | grep -v grep | awk '{ print \$2 }')


/opt/acmefetch/bin/acmefetch --cfg=/opt/acmefetch/etc/acmefetch.cfg

# stop http-server
kill \$http_pid

# at the first run pveproxy-ssl.key don't excist
if [[ ! -e /etc/pve/local/pveproxy-ssl.key ]]
then
  touch /etc/pve/local/pveproxy-ssl.key
fi

if [[ \$(diff -q /etc/ssl/private/pveproxy-ssl.key /etc/pve/local/pveproxy-ssl.key) ]]
then
  nodes=\$(ls /etc/pve/nodes)
  for i in \$nodes
  do
  cp -p /etc/ssl/private/pveproxy-ssl.key /etc/pve/nodes/\$i/
  cat /etc/ssl/certs/pveproxy-ssl.pem > /etc/pve/nodes/\$i/pveproxy-ssl.pem
  cat /etc/ssl/certs/pveproxy-ssl-chain.pem >> /etc/pve/nodes/\$i/pveproxy-ssl.pem
  ssh \$i service pveproxy restart
  done
fi
EOF

chmod +x /usr/local/sbin/le_cert_renew.sh
create and modify acmefetch config to fit for your systems (accountKeyPath + commonName + sites)
- the forum eat spaces, which are important for the config-file.
create the file first and after that edit the file for the right values:
Code:
cat /opt/acmefetch/etc/acmefetch.cfg
{
  "GENERAL": {
   "ACMEstaging": "acme-staging.api.letsencrypt.org",
   "ACMEservice": "acme-v01.api.letsencrypt.org",
   "accountKeyPath": "/etc/pve/.le/account.key"
  },
   "CERTS": [
       {
        "certOutput": "/etc/ssl/certs/pveproxy-ssl.pem",
        "certFormat": "PEM",
        "keyOutput": "/etc/ssl/private/pveproxy-ssl.key",
        "keyFormat": "PEM",
        "chainOutput": "/etc/ssl/certs/pveproxy-ssl-chain.pem",
        "chainFormat": "PEM",
        "commonName": "pve.example.com",
        "SITES": {
           "pve.example.com": {
              "challengeHandler": "LocalFile",
              "challengeConfig": {
              "www_root": "/opt/acmefetch/www/",
              }
          },
              "pve-a.example.com": {
                "challengeHandler": "LocalFile",
                "challengeConfig": {
                "www_root": "/opt/acmefetch/www/",
                }
              },
              "pve-b.example.com": {
                "challengeHandler": "LocalFile",
                "challengeConfig": {
                "www_root": "/opt/acmefetch/www/",
                 }
              },
               "pve-c.example.com": {
                  "challengeHandler": "LocalFile",
                 "challengeConfig": {
                 "www_root": "/opt/acmefetch/www/",
                 }
              },
          }
      }
  ]
}
extend HAproxy config on your haproxy-server - first the part for LetsEncrypt:
Code:
frontend http_frontend
  bind *:80
  mode http
  acl acl_pve  hdr(host) -i pve.example.com
  acl acl_pve  hdr(host) -i pve-a.example.com
  acl acl_pve  hdr(host) -i pve-b.example.com
  acl acl_pve  hdr(host) -i pve-c.example.com
  use_backend pve-http  if acl_pve

backend pve-http
  mode http
  server pve 192.168.200.11:8080
Code:
service haproxy reload
On the first node start the webserver to control the external access
Code:
cd /opt/acmefetch/www
python3 -m http.server 8080
from external must be following appear in an browser with pve.example.com, pve-a.example.com and so on:
Code:
  Hello.
  The python httpd server is working.
  and waiting for letsencrypt
If all right stop the web-server with <cntl>-C

Lets start
Code:
/usr/local/sbin/le_cert_renew.sh
 
Last edited:
To access the pve-gui from outside use a config like this in haproxy.cfg on your proxy-host:
Code:
frontend pve_frontend
  bind *:8006
  mode tcp
  option httpclose
  option forwardfor
  reqadd X-Forwarded-Proto:\ https
  default_backend pve-server
  timeout client  65000

backend pve-server
  mode tcp
  balance roundrobin
  timeout server  65000
  stick-table type ip size 200k expire 60m
  stick on src
  server pve-a 192.168.200.11:8006
  server pve-b 192.168.200.12:8006
  server pve-c 192.168.200.13:8006