[SOLVED] Proxmox borked after reboot - webui/ssh at first and now cert errors (certificate verify failed)

Stephen Smith

Member
Apr 25, 2018
4
0
6
29
Hi, so we had an unexpected power outage this morning and found that none of our VMs are coming back up. Upon further inspection, it seems the proxmox web ui doesn't load (hangs after Accept */*). Proxmox was recently updated to the latest version (2 major versions) and I thought I thoroughly rebooted after all updates finished to make sure it would recover in the event of a power outage (and it typically does) but something must have been overlooked...

Code:
*   Trying 192.168.1.3:8006...
* Connected to 192.168.1.3 (192.168.1.3) port 8006 (#0)
> GET / HTTP/1.1
> Host: 192.168.1.3:8006
> User-Agent: curl/7.78.0
> Accept: */*
>

As well as getting pubkey denied when trying to ssh in.

I checked journalctl -u pve-cluster -b after googling around for "web ui not loading", and saw

Code:
proxmox pmxcfs[1082]: [main] crit: unable to get local IP address

and suggestions to fix the hostname, so I checked /etc/hosts and /etc/hostname, my machine's hostname is "proxmox"

Code:
root@proxmox:~# hostname -f
proxmox.home.stephen304.com
root@proxmox:~# hostname
proxmox

/etc/hosts:

Code:
127.0.0.1 localhost.localdomain localhost
192.168.1.3 proxmox.home.stephen304.com pve pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I changed the second line to

Code:
192.168.1.3 proxmox.home.stephen304.com proxmox pve pvelocalhost

And now I can ssh in again and the web UI loads again. However all the VMs have a question mark icon and I can't seem to actually do any operations and I see this error everywhere:

Code:
Error Connection error 596: tls_process_server_certificate: certificate verify failed

Additional information, this is just a bare metal single node install. The last updates were applied maybe 2 weeks ago, and it's internal only so all certs that proxmox knows about should be self signed or whatever the default is.

Any help would be greatly appreciated, thanks!
 
Last edited:
Try running pvecm updatecerts --force
 
Try running pvecm updatecerts --force
I ran that:

Code:
root@proxmox:~# pvecm updatecerts -f
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts

and just to be sure

Code:
root@proxmox:/etc/pve/nodes# pvecm updatecerts --force
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts

Unfortunately it doesn't seem to have changed anything, and the certificate creation date in the certificates part of the web ui doesn't seem to have changed if that's supposed to. Should I try deleting certs before running that?
 
Last edited:
Just want to bump this as I still haven't figured out what's wrong with proxmox. Does anybody have any ideas on what I can check / what logs I should look at to narrow down the issue?

Thanks!

Edit: After clicking around the UI some and checking the logs, I saw this error if that helps:


Code:
Sep 18 23:27:01 proxmox systemd[1]: Finished Proxmox VE replication runner.
Sep 18 23:27:28 proxmox pveproxy[1120]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:27:34 proxmox pveproxy[1119]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:27:58 proxmox pveproxy[1121]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:28:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Sep 18 23:28:01 proxmox systemd[1]: pvesr.service: Succeeded.
Sep 18 23:28:01 proxmox systemd[1]: Finished Proxmox VE replication runner.
Sep 18 23:28:41 proxmox pveproxy[1120]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:29:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Sep 18 23:29:01 proxmox pveproxy[1121]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:29:01 proxmox systemd[1]: pvesr.service: Succeeded.
Sep 18 23:29:01 proxmox systemd[1]: Finished Proxmox VE replication runner.
Sep 18 23:29:15 proxmox pveproxy[1119]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:29:46 proxmox pveproxy[1120]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:30:00 proxmox systemd[1]: Starting Proxmox VE replication runner...
Sep 18 23:30:01 proxmox systemd[1]: pvesr.service: Succeeded.
Sep 18 23:30:01 proxmox systemd[1]: Finished Proxmox VE replication runner.
Sep 18 23:30:01 proxmox pveproxy[1121]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Sep 18 23:30:20 proxmox pveproxy[1119]: Could not verify remote node certificate '07:51:14:29:5F:39:05:49:37:93:73:94:62:9F:0A:1B:9F:FC:7F:49:BF:41:AF:4C:DB:F7:41:73:5A:13:29:C3' with list of pinned certificates, refreshing cache
Logs

This is just a single node setup so I guess remote node just means the local system
 
Last edited:
Actually it seems I've figured it out - I noticed that in /etc/pve/nodes/pve/qemu-server there were .conf files for every VM I run. But the hostname is proxmox not pve, so I moved the conf files to the /etc/pve/nodes/proxmox/qemu-server folder and rebooted and all the VMs came up. I'm not sure how this mismatch happened since I only followed upgrade instructions from the wiki, maybe something in the upgrade path defaulted to pve.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!