Added SSL certs, now cannot access web GUI

[cwoelkers]

I'm running a cluster of four Proxmox community servers, Proxmox 7.3-3 on Debian 11, in a cluster. I was getting tired of the SSL warnings from the self-signed certificate so created a CSR on each and signed them on my local CA server, I'm aware of ACME but did not want to use an external service for a private internal page, as I've done for my other local non-Proxmox servers. Once I uploaded the keys and certs things seemed to work great and I had no issues. Then I left it untouched, as I didn't need to do anything, for about a week and now I cannot access three of the four servers. When I try to access them using Chrome I get a "This site can't be reached" "ERR_CONNECTION_CLOSED" message. When using Firefox I get a "Secure Connection Failed" "PR_END_OF_FILE_ERROR" message. Within the one server web GUI that I can login to the others all show as running but I get "Broken Pipe (596)" for all of the configuration pages.

I have tried restarting the pveproxy service and while that generated no errors itself it did not fix anything. A system restart has been considered but that is a last resort due to the running VMs. My own guess is that the installation of the SSL cert caused an issue but why would it work initially and then fail?

 

[Chris]

I'm running a cluster of four Proxmox community servers, Proxmox 7.3-3 on Debian 11, in a cluster. I was getting tired of the SSL warnings from the self-signed certificate so created a CSR on each and signed them on my local CA server, I'm aware of ACME but did not want to use an external service for a private internal page, as I've done for my other local non-Proxmox servers. Once I uploaded the keys and certs things seemed to work great and I had no issues. Then I left it untouched, as I didn't need to do anything, for about a week and now I cannot access three of the four servers. When I try to access them using Chrome I get a "This site can't be reached" "ERR_CONNECTION_CLOSED" message. When using Firefox I get a "Secure Connection Failed" "PR_END_OF_FILE_ERROR" message. Within the one server web GUI that I can login to the others all show as running but I get "Broken Pipe (596)" for all of the configuration pages.

I have tried restarting the pveproxy service and while that generated no errors itself it did not fix anything. A system restart has been considered but that is a last resort due to the running VMs. My own guess is that the installation of the SSL cert caused an issue but why would it work initially and then fail?

Hi,
a few things to consider:

• Is your root certificate still valid?
• Was your certificate installed correctly? See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_certificate_management and check the respective file contents
• What is the output of curl -v https://<domain-or-ip>:8006/?

 

[cwoelkers]

Funny, I am also a Chris. Guess we really are everywhere. Anyways...

The root certificate is valid through 2025 so no worries there.
I installed the certificate using the web GUI by under System > Certificates for each host using the "Upload Custom Certificate" button.
Here is the requested curl output for one of the nodes, the others are the same.

curl -v https://vhost2:8006/
*   Trying 192.168.0.22...
* TCP_NODELAY set
* Connected to vhost2 (192.168.0.22) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vhost2:8006
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vhost2:8006

Looks like it wants a ca-bundle, something I did not add even on the one that still works. Though it wants it under /etc/pki which does not exist on any of the systems, might just be a default location for it though.

Chris

 

[Chris]

Funny, I am also a Chris. Guess we really are everywhere. Anyways...

The root certificate is valid through 2025 so no worries there.
I installed the certificate using the web GUI by under System > Certificates for each host using the "Upload Custom Certificate" button.
Here is the requested curl output for one of the nodes, the others are the same.

curl -v https://vhost2:8006/
*   Trying 192.168.0.22...
* TCP_NODELAY set
* Connected to vhost2 (192.168.0.22) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: vhost2:8006
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to vhost2:8006

Looks like it wants a ca-bundle, something I did not add even on the one that still works. Though it wants it under /etc/pki which does not exist on any of the systems, might just be a default location for it though.

Chris

Yes, seems like we are everywhere

This is somehow odd, some further things you can check:

• Check the pveproxy logs for errors, journalctl -b -u pveproxy.service
• Check the certificates and keys located at /etc/pve/local/pveproxy-ssl.pem and pveproxy-ssl.key for each of the node. These files should contain your custom certificates/keys. Check also the output of pvenode cert info on each node.
• If all the above seems to be fine, you might consider removing and reinstalling the certificates in oder to check if that solves the issue. Make sure to restart pveproxy after that (the WebUI will do this if you do it from there, otherwise pvenode cert delete 1 will remove the cert and restart pveproxy on the node).
• A quick online search suggests that cipher incompatibility might lead to such an error message, see https://github.com/openssl/openssl/issues/9566 . Whats is your pveversion -v and openssl version?

 

[cwoelkers]

I figured it out. The certs I generated initially did not have the cluster interface IP defined as a Subject Alternative Name, just the main IP and the hostname. Once I added it everything worked fine.

 

[area51x]

I figured it out. The certs I generated initially did not have the cluster interface IP defined as a Subject Alternative Name, just the main IP and the hostname. Once I added it everything worked fine.

Can you please explain this further. I'm also unable to sign into my web ui and running the curl directly on the server with a direct attached keyboard gave me a cert error. FYI I don't have a cluster.

 

[Chris]

Can you please explain this further. I'm also unable to sign into my web ui and running the curl directly on the server with a direct attached keyboard gave me a cert error. FYI I don't have a cluster.

Hi,
please provide the exact error messages you get. Does this happen after you installed a custom certificate? If so, how did you install the certificate?

 

[cwoelkers]

Can you please explain this further. I'm also unable to sign into my web ui and running the curl directly on the server with a direct attached keyboard gave me a cert error. FYI I don't have a cluster.

I do have a cluster and have defined a separate network interface for corosync to handle that traffic which exists on our management VLAN(not VLAN 1). As I wanted access to the web interface on that VLAN I needed to add the interface's IP address to the SSL cert before everything worked.

 

[area51x]

Hi,
please provide the exact error messages you get. Does this happen after you installed a custom certificate? If so, how did you install the certificate?

No I didn't install any custom certificates.

 

[Chris]

No I didn't install any custom certificates.

View attachment 59410

If you have a self signed certificate, you will have to use the -k flag in curl to accept the insecure connection, e.g. curl -v -k https://localhost:8006. You can also inspect the certificate currently used by the Proxmox VE host by running pvenode cert info. This will show you relevant information such as issuer, san, fingerprint validity date ecc.

You can force to regenerate the certificate via pvecm updatecerts -f.

I'm also unable to sign into my web ui

You only showed the output of the curl command, what error do you get when trying to log in to the server? Do you use the correct realm for your user account? Do you have 2FA enabled?

You should check the output of systemctl status pveproxy.service and cat /var/log/pveproxy/access.log.

 

[area51x]

If you have a self signed certificate, you will have to use the -k flag in curl to accept the insecure connection, e.g. curl -v -k https://localhost:8006. You can also inspect the certificate currently used by the Proxmox VE host by running pvenode cert info. This will show you relevant information such as issuer, san, fingerprint validity date ecc.

You can force to regenerate the certificate via pvecm updatecerts -f.


You only showed the output of the curl command, what error do you get when trying to log in to the server? Do you use the correct realm for your user account? Do you have 2FA enabled?

You should check the output of systemctl status pveproxy.service and cat /var/log/pveproxy/access.log.

This site can’t be reached​

192.168.1.209 refused to connect.


Try:

• Checking the connection
• Checking the proxy and the firewall

ERR_CONNECTION_REFUSED

 

[area51x]

The curl with -k from my laptop gives:

* Trying 192.168.1.209:8006...
* connect to 192.168.1.209 port 8006 failed: Connection refused
* Failed to connect to 192.168.1.209 port 8006 after 9 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to 192.168.1.209 port 8006 after 9 ms: Couldn't connect to server

Directly from the server I can ping apple.com.

---

systemctl status pveproxy.service gives no errors - everything is active

 

[Chris]

The curl with -k from my laptop gives:

* Trying 192.168.1.209:8006...
* connect to 192.168.1.209 port 8006 failed: Connection refused
* Failed to connect to 192.168.1.209 port 8006 after 9 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to 192.168.1.209 port 8006 after 9 ms: Couldn't connect to server

Directly from the server I can ping apple.com.

---

systemctl status pveproxy.service gives no errors - everything is active

Well, this sounds more like a network issue rather than a login error:

1. Are you able to ping the Proxmox VE host?
2. Is there a firewall potentially blocking your connections?
3. Can you ssh into the host?

 

[pkr]

has this been fixed, i am running into similar issues, and the certificates seems to be valid until 2025

 

[tawfiq.rm]

pvenode cert set /etc/pve/local/pve-ssl.pem /etc/pve/local/pve-ssl.key

solved my issue

 

[joshualhopkins]

Thank you @tawfiq.rm. This saved my bacon. One thing to add, I did have to add the -f at the end to force it. Thanks so much for the assist.