no webgui today

[kwauters]

I've changed nothing but I cannot access the webgui any more, the virtual machines are running and I can SSH into proxmox.
Trying to restart the service gives me errors.

service pve-cluster restart
I've run some command and following failed
Job for pve-cluster.service failed because the control process exited with error code.
See "systemctl status pve-cluster.service" and "journalctl -xe" for details.

systemctl status pve-cluster.service
pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor p>
Active: failed (Result: exit-code) since Thu 2022-01-13 21:18:28 CET; 58s >
Process: 687055 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTIO>
CPU: 8ms

journalctl -xe
Jan 12 04:38:37 pve rsyslogd[760]: action 'action-1-builtinmfile' (module 'builtinmfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Jan 12 04:38:37 pve rsyslogd[760]: file '/var/log/syslog'[9] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: Read-only file system [v8.2102.0 try https://www.rsyslog.com/e/2027 ]
Jan 12 04:38:37 pve rsyslogd[760]: action 'action-1-builtinmfile' (module 'builtinmfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https://www.rsyslog.com/e/2027 ]

The error code indicates an IO error but I'm lost to solve this problem.

 

[Sourcenux]

Your root-filesystem seems to be readonly - You may check if `dmesg` contains any further FS or IO-Errors which have caused the FS to go readonly in the first place.

 

[kwauters]

Thank you, "dmesg" turned my screen red with error codes, "fsck" and a reboot fixed the problem.

 

[t.lamprecht]

Did you have a power outage (and no enterprise disks with powerloss protection) recently?
As else that could mean that one or more disks are getting bad, at least partially. I'd definitively observer that and ensure that you got backup (and tested restoring them) - which is natually a good idea in general.

 

[kwauters]

No power outage that I'm aware of.
I only have only one node so I made an image from the disk after fixing the problem and I'm currently running from the new disk.
Keeping the image and the original disk as a backup since I don't change the setup often.

Special thanks to Sourcenux and t.lamprecht for their help.