Can’t Access the Web GUI Issue

[emon]

Hi, I was going to create a cluster and then my system crashed. I can SSH but no GUI or VMs are running.

Output of systemctl status pve-cluster pveproxy

     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sat 2022-07-02 19:15:33 +06; 17min ago
    Process: 1489 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 12ms

Jul 02 19:15:33 pve2 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Jul 02 19:15:33 pve2 systemd[1]: Stopped The Proxmox VE cluster filesystem.
Jul 02 19:15:33 pve2 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Jul 02 19:15:33 pve2 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Jul 02 19:15:33 pve2 systemd[1]: Failed to start The Proxmox VE cluster filesystem.

● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-07-02 19:15:35 +06; 17min ago
    Process: 1495 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 1496 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
    Process: 1829 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
   Main PID: 1497 (pveproxy)
      Tasks: 4 (limit: 38437)
     Memory: 135.3M
        CPU: 38.507s
     CGroup: /system.slice/pveproxy.service
             ├─1497 pveproxy
             ├─2333 pveproxy worker
             ├─2334 pveproxy worker
             └─2335 pveproxy worker

Jul 02 19:33:04 pve2 pveproxy[2333]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1917.
Jul 02 19:33:05 pve2 pveproxy[2331]: worker exit
Jul 02 19:33:05 pve2 pveproxy[2332]: worker exit
Jul 02 19:33:05 pve2 pveproxy[1497]: worker 2332 finished
Jul 02 19:33:05 pve2 pveproxy[1497]: worker 2331 finished
Jul 02 19:33:05 pve2 pveproxy[1497]: starting 2 worker(s)
Jul 02 19:33:05 pve2 pveproxy[1497]: worker 2334 started
Jul 02 19:33:05 pve2 pveproxy[1497]: worker 2335 started
Jul 02 19:33:05 pve2 pveproxy[2334]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1917.
Jul 02 19:33:05 pve2 pveproxy[2335]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1917.

Output of journalctl -xe

░░
░░ A start job for unit corosync.service has finished successfully.
░░
░░ The job identifier is 1838.
Jul 02 19:37:24 pve2 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ Automatic restarting of the unit pve-cluster.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Jul 02 19:37:24 pve2 systemd[1]: Stopped The Proxmox VE cluster filesystem.
░░ Subject: A stop job for unit pve-cluster.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A stop job for unit pve-cluster.service has finished.
░░
░░ The job identifier is 1916 and the job result is done.
Jul 02 19:37:24 pve2 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Jul 02 19:37:24 pve2 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit pve-cluster.service has entered the 'failed' state with result 'exit-code'.
Jul 02 19:37:24 pve2 systemd[1]: Failed to start The Proxmox VE cluster filesystem.
░░ Subject: A start job for unit pve-cluster.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pve-cluster.service has finished with a failure.
░░
░░ The job identifier is 1916 and the job result is failed.
Jul 02 19:37:24 pve2 systemd[1]: Condition check resulted in Corosync Cluster Engine being skipped.
░░ Subject: A start job for unit corosync.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit corosync.service has finished successfully.
░░
░░ The job identifier is 1994.
Jul 02 19:37:27 pve2 pveproxy[3525]: worker exit
Jul 02 19:37:27 pve2 pveproxy[1497]: worker 3525 finished
Jul 02 19:37:27 pve2 pveproxy[1497]: starting 1 worker(s)
Jul 02 19:37:27 pve2 pveproxy[1497]: worker 3558 started
Jul 02 19:37:27 pve2 pveproxy[3558]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1917.
Jul 02 19:37:27 pve2 pveproxy[3526]: worker exit
Jul 02 19:37:27 pve2 pveproxy[3527]: worker exit
Jul 02 19:37:27 pve2 pveproxy[1497]: worker 3527 finished
Jul 02 19:37:27 pve2 pveproxy[1497]: worker 3526 finished
Jul 02 19:37:27 pve2 pveproxy[1497]: starting 2 worker(s)
Jul 02 19:37:27 pve2 pveproxy[1497]: worker 3559 started
Jul 02 19:37:27 pve2 pveproxy[1497]: worker 3560 started
Jul 02 19:37:27 pve2 pveproxy[3559]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1917.
Jul 02 19:37:27 pve2 pveproxy[3560]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1917.

pvecm status

root@pve2:/var/lib/pve-cluster# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

 

[emon]

Surprisingly nothing at /etc/pve also

 

[Dunuin]

Surprisingly nothing at /etc/pve also

Thats not surprising. /etc/pve is the mountpoint of a special pmxcfs filesystem and not a normal folder. If your pve-cluster service isn't working, there is nothing mounted.

 

[emon]

@Dunuin I got that after lots of web scraping. I have /var/lib/pve-cluster/config.db which has all of my existing VMS configs but with that, I can't get my pve-cluster.service working. So what I have done mv config.db config.db.bk then rebooted the server. Everything was working but no VMS. Although I can see the volumes of my VMS in local storage. When I tried mv config.db.bk config.db and rebooted the same issue again. I just can't find a way to get my previous VMS working now.

 

[emon]

I have tried pmxcfs -l with my previous config.db, here's what I'm getting.

[database] crit: found entry with duplicate name 'qemu-server' - A:(inode = 0x000000000044E354, parent = 0x000000000044E353, v./mtime = 0x44E354/0x1656753721) vs. B:(inode = 0x000000000044E596, parent = 0x000000000044E353, v./mtime = 0x44E596/0x1656754674)
[database] crit: DB load failed
[main] crit: memdb_open failed - unable to open database '/var/lib/pve-cluster/config.db'
[main] notice: exit proxmox configuration filesystem (-1)

 

[Neobin]

No experience with this, but read about this error recently. Maybe it helps:
https://forum.proxmox.com/threads/hostname-changed-now-nodes-gone-from-etc-pve.111607/#post-481503

 

[emon]

Thanks, mate for the effort. My scenario was super urgent. I have waited but none responded with a proper fix. I have reset the whole cluster and the VMS. That was a lot of work.