[SOLVED] Renewing Certificate Broke Web UI

[jamesaepp]

Single Proxmox VE system - no clustering. I don't remember the exact version but there were no pending updates, I can say that much.

I use pfSense as my root CA for my homelab. Proxmox is one of the small number of systems I issue certs for. There's no intermediate CA.

About 10 months ago I installed the first/inaugural certificate for this PVE system and all was fine. It's coming up for expiry so I decided to spend some time working on that new certificate issuance. I generated a new certificate on the pfSense firewall with the below settings:

• Common Name + SAN (DNS) set to the FQDN of my PVE host
• Signature algorithm sha512RSA
• Key algorithm ECCDH_P256
• EKUs - Server auth, client auth

I think the original certificate was an RSA2048 cert so maybe I was too aggressive with turning up the dials on security.

I downloaded the new certificate as well as the root CA (in their base64 formats) and concatenated them together in one file for the public side. I then uploaded the text/base64 contents of both the public key/chain and the private key to the fields in the certificates section of the host in the web interface. After doing this - the web interface completely disappeared and hasn't returned.

All I get in my browser (MS Edge) is "It looks like <FQDN> closed the connection". Wireshark shows the TCP 3-way handshake works fine, then my client sends the TLS 1.2 client hello, then ACK/Fin-ack messages go out bidirectionally on both sides (started by the PVE host).

SSHing to the host and running journalctl shows this - is it normal for the workers to exit or is it just crashing?

Oct 14 19:11:22 cvhost0x00 systemd[1]: Reloading pveproxy.service - PVE API Proxy Server...
░░ Subject: A reload job for unit pveproxy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A reload job for unit pveproxy.service has begun execution.
░░
░░ The job identifier is 6764.
Oct 14 19:11:26 cvhost0x00 pveproxy[2014864]: send HUP to 930
Oct 14 19:11:26 cvhost0x00 pveproxy[930]: received signal HUP
Oct 14 19:11:26 cvhost0x00 pveproxy[930]: server closing
Oct 14 19:11:26 cvhost0x00 pveproxy[930]: server shutdown (restart)
Oct 14 19:11:26 cvhost0x00 systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
░░ Subject: A reload job for unit pveproxy.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A reload job for unit pveproxy.service has finished.
░░
░░ The job identifier is 6764 and the job result is done.
Oct 14 19:11:31 cvhost0x00 pveproxy[930]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Oct 14 19:11:31 cvhost0x00 pveproxy[930]: restarting server
Oct 14 19:11:31 cvhost0x00 pveproxy[930]: starting 3 worker(s)
Oct 14 19:11:31 cvhost0x00 pveproxy[930]: worker 2014890 started
Oct 14 19:11:31 cvhost0x00 pveproxy[930]: worker 2014891 started
Oct 14 19:11:31 cvhost0x00 pveproxy[930]: worker 2014892 started
Oct 14 19:11:36 cvhost0x00 pveproxy[930]: worker 1832116 finished
Oct 14 19:11:36 cvhost0x00 pveproxy[930]: worker 1832114 finished
Oct 14 19:11:36 cvhost0x00 pveproxy[930]: worker 1832115 finished
Oct 14 19:11:40 cvhost0x00 pveproxy[2014899]: worker exit
Oct 14 19:11:40 cvhost0x00 pveproxy[2014900]: worker exit
Oct 14 19:11:40 cvhost0x00 pveproxy[2014901]: worker exit
Oct 14 19:14:26 cvhost0x00 sshd[2015323]: Accepted password for root from 10.29.10.5 port 65454 ssh2

I rebooted the host but that didn't help - web UI still broken. I would prefer to troubleshoot this (for a bit) before I consider rebuilding the host.

 

[jamesaepp]

?PARTIAL? SOLUTION

This was human error on my part - my mistake was starting the new PEM chain with the root CA certificate before the PEM chain for the end-entity certificate (I genuinely thought that was correct form, turns out not).

What worked for me was roughly this:

1. SSH to the PVE host (I used root account, I don't know what minimal perms are required)
2. `cd /etc/pve/nodes/<YourNodeName>/`
3. Edit the file `pveproxy-ssl.pem` and correct the issue with the cert chain.
4. `smartctl restart pveproxy`
5. Test

If anyone has greater insight if there's anything else "behind the curtain" that should also be done, please comment. Things look in order from a glance at how the cert appears in the web UI .

 

[norbertk]

Sorry for a dumb question, but what does that mean : Edit the file `pveproxy-ssl.pem` and correct the issue with the cert chain. . What is to be done here ?

Thanks a lot for providing as solution to your own question. Much appreciated !

 

[jamesaepp]

Essentially what I mean is - a certificate file (in PEM format) looks something like the below (in plain text, easily opened in a text editor).

--- BEGIN CERTIFICATE ---
<base64 encoded data>
--- END CERTIFICATE ---

When you have a certificate "chain" multiple certificates are appended one after another like this:

--- BEGIN CERTIFICATE ---
<base64 encoded data>
--- END CERTIFICATE ---
--- BEGIN CERTIFICATE ---
<base64 encoded data>
--- END CERTIFICATE ---
(until completion)

So all I mean by "Edit the file `pveproxy-ssl.pem` and correct the issue with the cert chain" is to use nano/vi/whatever on the file in question and manipulate the file so that the chain is in the correct order (end-entity/subject certificate first, then issuing CA certificates if any, then root CA certificate).