Let's encrypt on a multi-node cluster

[holr]

Hello,
I have been following the instructions at https://pve.proxmox.com/wiki/Certificate_Management on a 5 node Proxmox cluster. Let's encrypt (using ACME) was used on the first node, PVE1, with great success. Accessing the server shows a valid certificate. Accessing VMs on PVE1 via noVNC and SPICE work.

If I access a VM hosted on one of the other servers (PVE2, PVE3, PVE4, PVE5), noVNC works. Unfortunately, SPICE (via remote-viewer) does not (it used to previously, with the default self-signed certs). The error that now appears in the console is:

(remote-viewer:6410): Spice-WARNING **: 00:19:17.047: ssl_verify.c:479penssl_verify: Error in server certificate verification: unable to get local issuer certificate (num=20:depth0:/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=pve2.xxxx.local)

(remote-viewer:6410): GSpice-WARNING **: 00:19:17.048: main-1:0: SSL_connect: error:00000001:lib(0):func(0):reason(1)

As the error suggests, I think this is because nodes PVE2-PVE5 do not have the Let's Encrypt certificates.

I am a little unclear on what to copy from PVE1 to PVE2-PVE5 to fix the certificate issue to get SPICE working again.

https://pve.proxmox.com/wiki/Certificate_Management mentions "For options 2 and 3 the file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pveproxy-ssl.key, which needs to be without password) is used."

Does this mean I must copy, from PVE1, /etc/pve/local/pveproxy-ssl.pem and /etc/pve/local/pveproxy-ssl.key to the same location (/etc/pve/local) on each of PVE2-PVE5 to install the Let's encrypt certs across the cluster?

Thank you!

 

[fabian]

spice does not use the pveproxy certificate, but the regular PVE generated self-signed certificate in /etc/pve/local/pve-ssl.pem, with the CA in /etc/pve/pve-root-ca.pem. did you touch either of those files?

openssl verify -CAfile /etc/pve/pve-root-ca.pem /etc/pve/local/pve-ssl.pem

should print

/etc/pve/local/pve-ssl.pem: OK

 

[holr]

====
*UPDATE*
I restored the backups of the pve-ssl.pem and pve-root-ca.pem on PVE1, which had the knock on effect of synching the correct pve-root-ca.pem across the cluster. All nodes now report OK after running

openssl verify -CAfile /etc/pve/pve-root-ca.pem /etc/pve/local/pve-ssl.pem

SPICE works again on PVE2-PVE5 (I confirmed by running a vm with SPICE from each server), but now SPICE has stopped working on PVE1! I cannot start a VM that has SPICE enabled, I receive the error:

kvm: warning: Spice: reds.c:2956:reds_init_ssl: Could not use private key file
kvm: failed to initialize spice server
TASK ERROR: start failed: QEMU exited with code 1

I tried restarting spiceproxy with

spiceproxy restart

but this didn't clear things. Any thoughts?

====
Thank you Fabian,
I ran
openssl verify -CAfile /etc/pve/pve-root-ca.pem /etc/pve/local/pve-ssl.pem
on each of the nodes.
It returned OK on PVE1, but VERIFICATION FAILED on PVE2-PVE5.

In the process of adding let's encrypt, I must have messed something up! Could you please suggest what I should do to fix it? Do I copy the /etc/pve/local/pve-ssl.pem and /etc/pve/pve-root-ca.pem from PVE1 to each of the other nodes?

Thank you!

 

[badji]

Bonjour,
I advise you to create a ct or vm in HA with Nginx, Haproxy or Caddy projects as Reverse-proxy with a letsencrypt certificate for your domain.
it will be the same certficate for all sub-domains ...
Here's a howto for Nginx on centos 8.1 for example :
https://www.cyberciti.biz/faq/configure-nginx-with-lets-encrypt-on-centos-8/

 

[fabian]

@holr please verify your cluster is actually in-sync and quorate. it sounds like pve-ssl.key got messed up on PVE1, you can regenerate it by deleting pve-ssl.pem and pve-ssl.key on PVE1, then running pvecm updatecerts on it.

 

[holr]

Thank you for the hint fabian,
I restored our archived pve-ssl.key file and et voila! SPICE is working again. Thank you!