Lost access to the GUI after SSL certificates deletion

I

Imtiazali

New Member
Dec 5, 2023
10
0
1
Good evening,
I lost access to the GUI Proxmox 8.1.4 after I deleted by mistakenly the following files : /etc/pve/local/pveproxy-ssl.pem and /etc/pve/local/pveproxy-ssl.key. I can still ssh to node but no GUI. below is the status of host file and pveproxy. Could you please guy guide me for the next steps. Thanks in advance,.


Code: 
root@tnoc:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
127.0.0.1 tnoc.miaj.tech tnoc
10.210.150.1 tnoc.miaj.tech tnoc
192.168.100.32 tnoc.miaj.tech tnoc
#10.210.150.1 tnoc.miaj.tech

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@tnoc:~#

Code: 
root@tnoc:~# pvecm updatecerts --force
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused


Code: 
 /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >lines 1-25/25 (END)...skipping...
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-01-24 18:23:25 +03; 10s ago
    Process: 3453 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 3454 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 3458 (pveproxy)
      Tasks: 4 (limit: 115332)
     Memory: 146.9M
        CPU: 974ms
     CGroup: /system.slice/pveproxy.service
             ├─3458 pveproxy
             ├─3475 "pveproxy worker"
             ├─3477 "pveproxy worker"
             └─3482 "pveproxy worker"

Jan 24 18:23:35 tnoc pveproxy[3464]: worker exit
Jan 24 18:23:35 tnoc pveproxy[3465]: worker exit
Jan 24 18:23:35 tnoc pveproxy[3475]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEv>
Jan 24 18:23:35 tnoc pveproxy[3458]: worker 3465 finished
Jan 24 18:23:35 tnoc pveproxy[3458]: worker 3464 finished
Jan 24 18:23:35 tnoc pveproxy[3458]: starting 2 worker(s)
Jan 24 18:23:35 tnoc pveproxy[3458]: worker 3477 started
Jan 24 18:23:35 tnoc pveproxy[3458]: worker 3482 started
Jan 24 18:23:35 tnoc pveproxy[3477]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEv>
Jan 24 18:23:35 tnoc pveproxy[3482]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEv>
 
Hi,
Imtiazali said:
root@tnoc:~# pvecm updatecerts --force ipcc_send_rec[1] failed: Connection refused ipcc_send_rec[2] failed: Connection refused ipcc_send_rec[3] failed: Connection refused Unable to load access control list: Connection refused
Click to expand...
These errors indicate that you probably do not have the proxmox cluster filesystem mounted correctly. Please check the status of the corresponding service via systemctl status pve-cluster.service. Is this a cluster node? If so, also check for quorum by running pvecm status

Only once the proxmox cluster filesystem is mounted correctly, the certificates can be accessed by the pveproxy, as these files are located on there.

Edit: Fixed typo
 
Last edited:
Chris said:
Hi,

These errors indicate that you probably do not have the proxmox cluster filesystem mounted correctly. Please check the status of the corresponding service via systemctl status pve-cluster.service. Is this a cluster node? If so, also check for quorum by running pvecm status

Only once the proxmox cluster filesystem is mounted correctly, the certificates can be accessed by the pveproxy, as these files are located on there.

Edit: Fixed typo
Click to expand...
Hi it's a standalone node. and here is the status of the service.


Code: 
root@tnoc:~# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
root@tnoc:~# systemctl status pve-cluster.service
× pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-01-24 18:23:26 +03; 19min ago
    Process: 3463 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 6ms

Jan 24 18:23:26 tnoc systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Jan 24 18:23:26 tnoc systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Jan 24 18:23:26 tnoc systemd[1]: pve-cluster.service: Start request repeated too quickly.
Jan 24 18:23:26 tnoc systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jan 24 18:23:26 tnoc systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.
root@tnoc:~#
 
As expected the cluster filesystem is not mounted correctly. Please check the systemd journal for the cause by running journalctl -u pve-cluster -b. Depending on what the cause of the failure is, it might be enough to restart the service via systemctl reset-failed pve-cluster.service && systemctl restart pve-cluster.service
 
Chris said:
As expected the cluster filesystem is not mounted correctly. Please check the systemd journal for the cause by running journalctl -u pve-cluster -b. Depending on what the cause of the failure is, it might be enough to restart the service via systemctl reset-failed pve-cluster.service && systemctl restart pve-cluster.service
Click to expand...

Here are the logs. /etc/pve is empty. tried to restart the cluster.service did a node reboot as well still the same.

root@tnoc:/etc/pve# journalctl -u pve-cluster -b Jan 24 17:51:18 tnoc systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem... Jan 24 17:51:18 tnoc pmxcfs[1808]: [main] notice: resolved node name 'tnoc' to '10.210.150.1' for default node IP address Jan 24 17:51:18 tnoc pmxcfs[1808]: [main] notice: resolved node name 'tnoc' to '10.210.150.1' for default node IP address Jan 24 17:51:18 tnoc pmxcfs[1808]: fuse: mountpoint is not empty Jan 24 17:51:18 tnoc pmxcfs[1808]: fuse: if you are sure this is safe, use the 'nonempty' mount option Jan 24 17:51:18 tnoc pmxcfs[1808]: [main] crit: fuse_mount error: File exists Jan 24 17:51:18 tnoc pmxcfs[1808]: [main] crit: fuse_mount error: File exists Jan 24 17:51:18 tnoc pmxcfs[1808]: [main] notice: exit proxmox configuration filesystem (-1) Jan 24 17:51:18 tnoc pmxcfs[1808]: [main] notice: exit proxmox configuration filesystem (-1) Jan 24 17:51:18 tnoc systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION Jan 24 17:51:18 tnoc systemd[1]: pve-cluster.service: Failed with result 'exit-code'. Jan 24 17:51:18 tnoc systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem. Jan 24 17:51:18 tnoc systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 1. Jan 24 17:51:18 tnoc systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem. Jan 24 17:51:18 tnoc systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem... Jan 24 17:51:18 tnoc pmxcfs[1959]: [main] notice: resolved node name 'tnoc' to '10.210.150.1' for default node IP address Jan 24 17:51:18 tnoc pmxcfs[1959]: [main] notice: resolved node name 'tnoc' to '10.210.150.1' for default node IP address Jan 24 17:51:18 tnoc pmxcfs[1959]: fuse: mountpoint is not empty Jan 24 17:51:18 tnoc pmxcfs[1959]: fuse: if you are sure this is safe, use the 'nonempty' mount option Jan 24 17:51:18 tnoc pmxcfs[1959]: [main] crit: fuse_mount error: File exists Jan 24 17:51:18 tnoc pmxcfs[1959]: [main] crit: fuse_mount error: File exists Jan 24 17:51:18 tnoc pmxcfs[1959]: [main] notice: exit proxmox configuration filesystem (-1) Jan 24 17:51:18 tnoc pmxcfs[1959]: [main] notice: exit proxmox configuration filesystem (-1) Jan 24 17:51:18 tnoc systemd[1]: pve-cluster.service: Control process exited, code=exited, status=255/EXCEPTION
 
Imtiazali said:
fuse: mountpoint is not empty
Click to expand...
Well, there seems to be something located under /etc/pve. Please check the output of ls -la /etc/pve. Also, check if there is no other filesystem mounted there via the output of mount | grep pve.
 
Last edited:
  • Like
Reactions: Imtiazali
Chris said:
Well, there seems to be something located under /etc/pve. Please check the output of ls -la /etc/pve. Also, check if there is no other filesystem mounted there via the output of mount | grep pve.
Click to expand...
Thank You so much there was a debug file. removed and restarted the service it's working normally now. I can access the GUI. Thanks again for pointing

Code: 
root@tnoc:~# cd /etc/pve/
root@tnoc:/etc/pve# ls
root@tnoc:/etc/pve# ls -la
total 10
drwxr-xr-x   2 root root   3 Jan 24 19:08 .
drwxr-xr-x 104 root root 205 Jan 24 19:11 ..
-rw-r--r--   1 root root   2 Jan 24 19:01 .debug
root@tnoc:/etc/pve# cat .debug
1
root@tnoc:/etc/pve# rm .debug
root@tnoc:/etc/pve# ls -la
total 9
drwxr-xr-x   2 root root   2 Jan 24 19:14 .
drwxr-xr-x 104 root root 205 Jan 24 19:11 ..
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom