can someone add 'adding intermediate certificates' to the cert wiki page?

[llamasales]

Hi guys,

I've spent ages bashing my head at bits of getting external-ca certificates working - the last hurdle turned out to be needing to append an intermediate CA cert to /etc/pve/pve-root-ca.pem.
After setting up my certificates (and intially get it wrong because I'd not followed the wiki well enough and touching the ones *not* to touch once i'd had initial problems...) I found I could not start one of my VMs - and it took checking /var/log/syslog or running 'qm start 100' to see the errors.

I was getting the dreaded errors of:

root@pve:~/pve# qm start 100
kvm: -vnc unix:/var/run/qemu-server/100.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
start failed: command '/usr/bin/kvm -id 100 ....<etc>'​

This was despite me following:
https://pve.proxmox.com/wiki/HTTPS_Certificate_Configuration_(Version_4.x_and_newer)
What fixed it was:
appending an active intermediate cert from letsencrypt.org/certificates/ to the end of /etc/pve/local/pve-ssl.pem

I guess its worth me admitting that i didn't actually use acme.sh - I'd already made the certs I wanted to use on my temporarily exposed webserver on a different host, so perchance my steps didn't replicate acme.sh exactly - but putting in a note mentioning the above at the end of the wiki page would likely save a lot of people a lot of headaches!

... Also - In my searching I'd found mention that letsencrypt features were to be somehow integrated into proxmox a year or two ago - did that end up happening and I just missed it?
Thanks

 

[fabian]

Hi guys,

I've spent ages bashing my head at bits of getting external-ca certificates working - the last hurdle turned out to be needing to append an intermediate CA cert to /etc/pve/pve-root-ca.pem.
After setting up my certificates (and intially get it wrong because I'd not followed the wiki well enough and touching the ones *not* to touch once i'd had initial problems...) I found I could not start one of my VMs - and it took checking /var/log/syslog or running 'qm start 100' to see the errors.

I was getting the dreaded errors of:

root@pve:~/pve# qm start 100
kvm: -vnc unix:/var/run/qemu-server/100.vnc,x509,password: Failed to start VNC server: Our own certificate /etc/pve/local/pve-ssl.pem failed validation against /etc/pve/pve-root-ca.pem: The certificate hasn't got a known issuer
start failed: command '/usr/bin/kvm -id 100 ....<etc>'​

This was despite me following:
https://pve.proxmox.com/wiki/HTTPS_Certificate_Configuration_(Version_4.x_and_newer)
What fixed it was:
appending an active intermediate cert from letsencrypt.org/certificates/ to the end of /etc/pve/local/pve-ssl.pem

I guess its worth me admitting that i didn't actually use acme.sh - I'd already made the certs I wanted to use on my temporarily exposed webserver on a different host, so perchance my steps didn't replicate acme.sh exactly - but putting in a note mentioning the above at the end of the wiki page would likely save a lot of people a lot of headaches!

... Also - In my searching I'd found mention that letsencrypt features were to be somehow integrated into proxmox a year or two ago - did that end up happening and I just missed it?
Thanks

you modified the wrong files because you did not follow the instructions.

if you want to add custom certificates

1. revert to the stock self-signed certificate state
2. follow the instructions to add custom certificates from a CA other than Let's Encrypt

or wait a little while longer, custom certificate upload via the GUI is coming soon

 

[llamasales]

Glad to hear it - i shouldve been clearer though - my above experience was after following the steps to revert & then following the other CA steps.

The VM did not start *after* following those steps. The wiki page instructions can still leave people with nonstarting vms with no clear reason; the instructions are inadequate. there are forum posts though of people saying ‘just concat the intermediate certs’ - which works - but the wiki really needs that info, which was super useful, even if just as a comment at the end... there are blog posts/other things showing a clear pattern of people having to use this workaround, but its a pretty bad experience that everyones just directed at the wiki page that doesnt mention how to fix the problem.

 

[fabian]

Glad to hear it - i shouldve been clearer though - my above experience was after following the steps to revert & then following the other CA steps.

if you had followed the instructions, you would not have the error.

there are two sets of certificate files, one is managed by PVE and not supposed to be touched (unless you want to completely re-create them via pvecm) - these are pve-root-ca.* (cluster-wide root CA) and pve-ssl.pem/.key (certificate and key, signed by root CA, on each node). the other is the optional key + certificate chain file on each node, stored in pveproxy-ssl.pem/.key. the latter can be added by the user, and if it is found, pveproxy will use it for the REST API / web interface.

The VM did not start *after* following those steps. The wiki page instructions can still leave people with nonstarting vms with no clear reason; the instructions are inadequate. there are forum posts though of people saying ‘just concat the intermediate certs’ - which works - but the wiki really needs that info, which was super useful, even if just as a comment at the end... there are blog posts/other things showing a clear pattern of people having to use this workaround, but its a pretty bad experience that everyones just directed at the wiki page that doesnt mention how to fix the problem.

the VM did not start because you modified the files you are not supposed to modify, instead of the ones you are supposed to.

 

[llamasales]

if you had followed the instructions, you would not have the error.

there are two sets of certificate files, one is managed by PVE and not supposed to be touched (unless you want to completely re-create them via pvecm) - these are pve-root-ca.* (cluster-wide root CA) and pve-ssl.pem/.key (certificate and key, signed by root CA, on each node). the other is the optional key + certificate chain file on each node, stored in pveproxy-ssl.pem/.key. the latter can be added by the user, and if it is found, pveproxy will use it for the REST API / web interface.



the VM did not start because you modified the files you are not supposed to modify, instead of the ones you are supposed to.

Hmm, ok, I can see you're right, but I really think the community's being done a disservice by the wikpage not better stressing the purpose of
/etc/pve/nodes/<node>/pve-ssl.pem|key
vs the optional (and not already present) files need to create, rather than replace existing keys (which is the intuitive thing people expect)
/etc/pve/nodes/<node>/pveproxy-ssl.pem|key

... there's many threads and even blog posts from people doing the workaround because they likely also didn't interpret what's going on correctly from that wiki page.

 

[fabian]

sorry, but the instructions list the correct paths in a copy-pastable manner and has the following note:

(make sure to use the correct certificate files and node!)

but like I said, certificate management over the API/CLI/GUI is already in the works.