[SOLVED] Web GUI not working, unknown issues after power outage

[tallguyjenks]

Im at my wits end. I've scoured the forum for solutions and no commands are helping and i have no idea what in the hell is wrong with the system and my knowledge of linux internals is not up to this task.

Ran

systemctl status pve-cluster pveproxy pvedaemon

pveversion -v

I've tried:

apt update
apt install --reinstall pve-manager proxmox-widget-toolkit
apt upgrade
apt full-upgrade
apt install pve-kernal-helper
systemctl restart pve-cluster.service

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

apt update

apt install --reinstall pve-manager proxmox-widget-toolkit

apt full-upgrade

 

[tallguyjenks]

apt install pve-kernal-helper

systemctl restart pve-cluster.service

 

[tallguyjenks]

I've also tried pieces of suggestions from these posts:

• https://forum.proxmox.com/threads/dpkg-hanging-when-upgrading-pve-kernel.95077/
• https://forum.proxmox.com/threads/failed-to-upgrade-packages.89089/
• https://forum.proxmox.com/threads/apt-update-errors-help.96527/
• https://forum.proxmox.com/threads/update-to-2-2-failed.110907/
• https://forum.proxmox.com/threads/n...c_send_rec-1-failed-connection-refused.41273/

And still i have no web GUI, my services WERE running despite the GUI earlier and now i don't even have that, and all this happened after a sudden blackout where my UPS didn't take over for some reason.

 

[bbgeek17]

and what does "journalctl -xe" say?
you can also try: pvecm updatecerts --force

you also appear to have some disk corruption and may need to run fsck on your drives.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[tallguyjenks]

and what does "journalctl -xe" say?
you can also try: pvecm updatecerts --force

you also appear to have some disk corruption and may need to run fsck on your drives.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

journalctl -xe

output is almost 1200 lines, but the red that keeps popping up is:

/etc/pve/local/pve-ss1.key: failed to load local private key (key_file or key) at /usr/share/per 15/PVE/APIServer/AnyEvent.pm line 1943.

pvecm updatecerts --force

yields:



i tried running fsck on a drive and I'm not sure what I'm doing with that command or how to repair the disk

 

[tallguyjenks]

working on it and making progress

running fsck on the disks seemed to work, i went a different route through the menus and they made more sense and actually wrote the changes so that worked

next i noticed that the problem of the SSL keys error seemed to be that the /etc/pve directory was not mounting and then that led to discovery that the /var/lib/pve-cluster/config.db was malformed and so i renamed with a .bak added to it and ran `pmxcfs -l` and that worked then ran `systemctl restart pve-cluster` and then checked `journalctl -xe` and the output looks a bit more promising:

 

[tallguyjenks]

ran `pvecm updatecerts` and it worked this time!
then ran `systemctl restart pve-cluster` but now the error is just

[main] crit: unable to acquire pmxcfs lock: Resource temporarily unavailable

 

[tallguyjenks]

reboot has fixed it! i have web GUI again

have to repair the /var/lib/pve-cluster/config.db file though as it looks like my node is now a blank slate and even though i have backups of the LXC's and VM's I'm not sure if my notes and other settings are accessible, anyone know anything to help me there?

 

[tallguyjenks]

looks like all i lost is a little configuration stuff like some storage settings and notes on the node object, restoring items from backups pretty much fixes the issue for me, calling this solved

 

[Pifouney]

@tallguyjenks hey, in yours firsts screenshots, when your're upgrading, you've a read-only SDB2 referenced. Reboot has re-mounted all in RW.

But your problem come of this read-only state. Check why your FS has gone read-only mode

 

[tallguyjenks]

@tallguyjenks hey, in yours firsts screenshots, when your're upgrading, you've a read-only SDB2 referenced. Reboot has re-mounted all in RW.

But your problem come of this read-only state. Check why your FS has gone read-only mode

thanks! turns out the disk was corrupted along with sdj2 from a power outage and my UPS not switching over >.>