Broken Proxmox by trying to add SSL Certificates

Aug 6, 2019
22
1
8
54
I recently made a complete mess of my Proxmox server (it's a homelab) by trying to install a GPU. Made such a mess that I gave up and reinstalled everything. This was fine until I tried to install the self-signed certificates that I'd had installed on the previous version of the server. (They were created with the Docker container division42llc/dotnet-webca)

I obviously did something wrong because now I can't access the web interface. Chrome and Firefox tell me the server closed the connection and for some reason Safari tells me I'm not connected to the internet.

I followed the instructions here

https://pve.proxmox.com/wiki/HTTPS_Certificate_Configuration_(Version_4.x,_5.0_and_5.1)

because I can't seem to find the 6.0 version with various searches. I did a search for .pem and .key and .ca and removed everything, restarted the server a few times but I'm in the same position. The server is working - all the VMs are up. I just can't get to the web interface.

Can someone tell me what else I need to delete and/or point me to the 6.0 version of the above wiki document?

Thanks,

Steve
 
That document is no use at all. The very first paragraph tells you it's not what I'm looking for. And to offer "I know this does not work at the moment, but if it would you could." as a suggestion makes me wonder what I pay my subscription for.
 
That document is no use at all. The very first paragraph tells you it's not what I'm looking for. And to offer "I know this does not work at the moment, but if it would you could." as a suggestion makes me wonder what I pay my subscription for.

that document is exactly what you should use - it migh thelp to read more than the first paragraph..

The REST API and web GUI are provided by the pveproxy service, which runs on each node.
You have the following options for the certificate used by pveproxy:
  1. By default the node-specific certificate in /etc/pve/nodes/NODENAME/pve-ssl.pem is used. This certificate is signed by the cluster CA and therefore not trusted by browsers and operating systems by default.
  2. use an externally provided certificate (e.g. signed by a commercial CA).
  3. use ACME (e.g., Let’s Encrypt) to get a trusted certificate with automatic renewal.
For options 2 and 3 the file /etc/pve/local/pveproxy-ssl.pem (and /etc/pve/local/pveproxy-ssl.key, which needs to be without password) is used.
Certificates are managed with the Proxmox VE Node management command (see the pvenode(1) manpage).

Do not replace or manually modify the automatically generated node certificate files in /etc/pve/local/pve-ssl.pem and /etc/pve/local/pve-ssl.key or the cluster CA files in /etc/pve/pve-root-ca.pem and /etc/pve/priv/pve-root-ca.key.

I suggest the following:
- remove /etc/pve/priv/pve-root-ca.key
- remove /etc/pve/pve-root-ca.pem
- remove /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.key /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pveproxy-ssl.key (on each node!)
- run 'pvecm updatecerts'
- run 'systemctl restart pveproxy'

now the web interface should work again, with the default self-signed certificates. now use the GUI or 'pvenode' to upload your certificate+key (it will be stored in '/etc/pve/local/pveproxy-ssl.pem' / '/etc/pve/local/pveproxy-ssl.key'
 
  • Like
Reactions: jtremblay and LinFJ
If you have a subscription with basic, standard or premium level you can contact the Support via Customer Portal.
At the customer portal your setup will be inspected in detail.
If you have a community-level subscription the support does not include a detailed failure analysis.
That document is no use at all.
See Fabian's answer.
 
- remove /etc/pve/priv/pve-root-ca.key
- remove /etc/pve/pve-root-ca.pem
- remove /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.key /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pveproxy-ssl.key (on each node!)
- run 'pvecm updatecerts'
- run 'systemctl restart pveproxy'
Thanks, fabian, very much! That saved my day after having destroyed my SSL/HTTPS connectivity by trying to upload my own certificates via WEB-GUI.
 
I'm also struggling to setup my certificate, so that there is ever only 1 private key to use for all my services. I would like to retrieve the cert once and then distribute it to all VMs and LXCs. In contrast to OP, I would like to have a wildcard cert, but the only difference here is the "Subject Alternative Name".

First I tried to issue a wildcard cert directly with the acme support in "Datacenter" but is sadly enough not supported... (Another Thread discussion)
Single domain works just perfect.

Next thing I tried is to upload a custom cert, as OP tired to do.
I issued a cert with certbot
Bash:
certbot certonly --manual --preferred-challenges=dns --email admin@example.com --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d *.example.com -d example.com
and got 4 files:
  • cert.pem
  • chain.pem
  • fullchain.pem (cert + chain)
  • privkey.pem
I uploaded fullchain and privkey and got the same problem as OP.

I compared the output of the proxmox acme cert (pveproxy-ssl.pem) and my certbot cert (fullchain.pem), the only difference being a missing new line between "cert" and "chain".

Code:
-----BEGIN CERTIFICATE-----
domain pubkey
-----END CERTIFICATE-----
******* MISSING NEW LINE *******
-----BEGIN CERTIFICATE-----
chain form let's encrypt
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
chain form let's encrypt
-----END CERTIFICATE-----

I'm not sure about specification if there needs to be a new line (=> certbot "bug") or if it can be ommitted (=> proxmox "bug").
Apache seems to not have no problem with the missing new line.

Wildcard support would be really nice to have.
 
does PVE accept the chain if you add the new line to the one generated by certbot?
 
then that's definitely something that should be improved! could you file an issue at https://bugzilla.proxmox.com with details (installed version, if possible the chain or the chain with just the server/leaf cert censored, ..)? thanks!
 
that document is exactly what you should use - it migh thelp to read more than the first paragraph..



I suggest the following:
- remove /etc/pve/priv/pve-root-ca.key
- remove /etc/pve/pve-root-ca.pem
- remove /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.key /etc/pve/local/pveproxy-ssl.pem /etc/pve/local/pveproxy-ssl.key (on each node!)
- run 'pvecm updatecerts'
- run 'systemctl restart pveproxy'

now the web interface should work again, with the default self-signed certificates. now use the GUI or 'pvenode' to upload your certificate+key (it will be stored in '/etc/pve/local/pveproxy-ssl.pem' / '/etc/pve/local/pveproxy-ssl.key'
I tried to import Certs I created in FreeIPA and borked the WebUI THIS SOLVED THE ISSUE! TY
 
I can't seem to get my company's purchased Sectigo wildcard ssl certificate into proxmox. I've used these same certs/key to secure Apache servers, firewall admin GUIs, server OOB management interfaces (ie, iLO, DRAC, and supermicro), and all our VmWare ESXi installations. Since I'm trying proxmox (8.2.2) for the first time on two servers in our lab after the latest VmWare/Broadcom baloney, I have been trying to get the wildcard certs installed on the two I have running before I go any farther. I have tried both the CLI upload with "pvenode cert set" as well as the GUI upload (from System->Certificates) of the bundled certificates and private key files, and in all cases, the proxmox GUI never comes back. I did notice the mention of the chainfile needing newlines before each BEGIN and manually made that change to my chained PEMs, but no change in behavior: after "systemctl restart pveproxy" there is nothing at all responding on port 8006.

It's simple to restore the gui; I just follow the remove/update/restart commands mentioned above and the unsecured GUI comes right back, but what am I missing to have proxmox using an actual browser-accepted SSL connection? I really just don't want to constantly hassle with accepting manually accepting things, and since the big bosses have paid for a real wildcard cert, I expect I'd be able to use it here, like everywhere else (admittedly everything seems to require a slightly different format of the certs/chains/keys and has a different method to begin to use them - I'm sure I'm just not doing one little thing here). What am I missing?

I'm following steps from both this thread (earlier versions of proxmox, I guess) but also have gone straight to the referenced https://pve.proxmox.com/wiki/Certificate_Management document too and I can't figure out what's wrong. journalctl logs seem to show me successful pveproxy starting up with the new files - but nothing after that:
```
Jul 07 23:22:24 proxmox8 systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Jul 07 23:22:25 proxmox8 pveproxy[490948]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Jul 07 23:22:25 proxmox8 pveproxy[490949]: starting server
```
I hesitated to resurrect this older thread but it has provided me the biggest clues so far, so figured it was the right place.
 
Just after sending, I thought I'd try a different browser, and firefox does display more helpful information than Chrome though:
An error occurred during a connection to proxmox8.velleros.com:8006. PR_END_OF_FILE_ERROR

Error code: PR_END_OF_FILE_ERROR

In case that tells anyone something... thanks for having a look.
 
that sounds like maybe the file has a weird format? is it PEM encoded and containing two certificates (your server's, and the intermediate CA certificate)?

you can inspect with a text editor, and openssl's `x509` command
 
  • Like
Reactions: dbharsh
that sounds like maybe the file has a weird format? is it PEM encoded and containing two certificates (your server's, and the intermediate CA certificate)?

you can inspect with a text editor, and openssl's `x509` command
To me it looks good in vi; however it has 3 certs - I think there are several intermediates given in my provided "ca-bundle" plus my own wildcard one at the top. I've used the exact same "combined certificate" file to secure an esxi GUI. The only thing I changed was that I added a linefeed/blank line in between the certs as mentioned much earlier in this thread, just in case that was still an issue. But either way, the reaction is the same here on proxmox. I will check out that openssl command; I'm not familiar.
 
Oh and also, one note that when using the CLI method of "pvenode cert set" - the output from that command looks perfect - it's a nice summary of the org name, CA info, dates, etc and they all match what I'd expect.
 
could you try connecting with "openssl s_client" and dump the handshake/certs after having set the custom cert (chain)?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!