Let's encrypt on a multi-node cluster

H

holr

Well-Known Member
Jan 4, 2019
33
1
48
55
Hello,
I have been following the instructions at https://pve.proxmox.com/wiki/Certificate_Management on a 5 node Proxmox cluster. Let's encrypt (using ACME) was used on the first node, PVE1, with great success. Accessing the server shows a valid certificate. Accessing VMs on PVE1 via noVNC and SPICE work.

If I access a VM hosted on one of the other servers (PVE2, PVE3, PVE4, PVE5), noVNC works. Unfortunately, SPICE (via remote-viewer) does not (it used to previously, with the default self-signed certs). The error that now appears in the console is:

(remote-viewer:6410): Spice-WARNING **: 00:19:17.047: ssl_verify.c:479:openssl_verify: Error in server certificate verification: unable to get local issuer certificate (num=20:depth0:/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=pve2.xxxx.local)

(remote-viewer:6410): GSpice-WARNING **: 00:19:17.048: main-1:0: SSL_connect: error:00000001:lib(0):func(0):reason(1)

As the error suggests, I think this is because nodes PVE2-PVE5 do not have the Let's Encrypt certificates.

I am a little unclear on what to copy from PVE1 to PVE2-PVE5 to fix the certificate issue to get SPICE working again.

https://pve.proxmox.com/wiki/Certificate_Management mentions "For options 2 and 3 the file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pveproxy-ssl.key, which needs to be without password) is used."

Does this mean I must copy, from PVE1, /etc/pve/local/pveproxy-ssl.pem and /etc/pve/local/pveproxy-ssl.key to the same location (/etc/pve/local) on each of PVE2-PVE5 to install the Let's encrypt certs across the cluster?

Thank you!
 
spice does not use the pveproxy certificate, but the regular PVE generated self-signed certificate in /etc/pve/local/pve-ssl.pem, with the CA in /etc/pve/pve-root-ca.pem. did you touch either of those files?

Code: 
openssl verify -CAfile /etc/pve/pve-root-ca.pem /etc/pve/local/pve-ssl.pem

should print

Code: 
/etc/pve/local/pve-ssl.pem: OK
 
  • Like
Reactions: holr
====
*UPDATE*
I restored the backups of the pve-ssl.pem and pve-root-ca.pem on PVE1, which had the knock on effect of synching the correct pve-root-ca.pem across the cluster. All nodes now report OK after running
Code: 
openssl verify -CAfile /etc/pve/pve-root-ca.pem /etc/pve/local/pve-ssl.pem

SPICE works again on PVE2-PVE5 (I confirmed by running a vm with SPICE from each server), but now SPICE has stopped working on PVE1! I cannot start a VM that has SPICE enabled, I receive the error:
Code: 
kvm: warning: Spice: reds.c:2956:reds_init_ssl: Could not use private key file
kvm: failed to initialize spice server
TASK ERROR: start failed: QEMU exited with code 1

I tried restarting spiceproxy with

Code: 
spiceproxy restart

but this didn't clear things. Any thoughts?

====
Thank you Fabian,
I ran
openssl verify -CAfile /etc/pve/pve-root-ca.pem /etc/pve/local/pve-ssl.pem
on each of the nodes.
It returned OK on PVE1, but VERIFICATION FAILED on PVE2-PVE5.

In the process of adding let's encrypt, I must have messed something up! Could you please suggest what I should do to fix it? Do I copy the /etc/pve/local/pve-ssl.pem and /etc/pve/pve-root-ca.pem from PVE1 to each of the other nodes?

Thank you!
 
Last edited:
@holr please verify your cluster is actually in-sync and quorate. it sounds like pve-ssl.key got messed up on PVE1, you can regenerate it by deleting pve-ssl.pem and pve-ssl.key on PVE1, then running pvecm updatecerts on it.
 
  • Like
Reactions: holr
Thank you for the hint fabian,
I restored our archived pve-ssl.key file and et voila! SPICE is working again. Thank you!
 
You must log in or register to reply here.
Top Bottom