Login failed for Web UI but not SSH (Backup job also did not run)

[promoxer]

1. After a few days of uptime, I cannot login to my PVE, it loads, but after entering the username + password, I get "Login failed. Please try again"
2. However, logging in via SSH using the same username + password works
3. My scheduled nightly backup job in PVE also did not run
4. It goes back to normal after I reboot PVE, but why is a reboot necessary?

Any idea how to troubleshoot?

 

[Dunuin]

Verify that you use the PAM realm and not the PVE realm when logging in.
Copy+Paste your password instead of typing it in to prevent typos.
Login via SSH and check with df -h if your root filesystem is full.
Check with systemctl status pvecluster.service if the cluster service is working and mounting the cluster.db to /etc/pve.
Check with pvesm status if any storage isn't available overwhelming pvestatd.

 

[promoxer]

1. Yes, using Linux PAM
2. I don't type passwords, Bitwarden does it for me
3. Filesystem is not full
4. I'm not running a cluster, the /etc/pve/ folder is accessible

 

[promoxer]

A reboot solved the problem, but why should that be necessary? It is the 2nd time in the last 2 weeks I'm facing it and it feels like it is going to be a regular thing, so I need to find out what is wrong.

 

[Kiyo]

Same happening to me. Comes back a day or so after reboot,

an 31 16:07:53 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error Jan 31 16:07:58 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error Jan 31 16:08:03 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error Jan 31 16:08:08 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error Jan 31 16:08:13 Ritsu pvescheduler[731114]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout Jan 31 16:08:13 Ritsu pvescheduler[731113]: replication: cfs-lock 'file-replication_cfg' error: got lock request timeout Jan 31 16:08:13 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error Jan 31 16:08:18 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error Jan 31 16:08:23 Ritsu pve-ha-lrm[5250]: unable to write lrm status file - unable to delete old temp file: Input/output error
Lots of this in `journactl -b`

root@Ritsu:~# dd if=/dev/zero of=/etc/pve/testfile bs=4M count=1 dd: failed to open '/etc/pve/testfile': Input/output error

 

[Kiyo]

It's also affecting things like pct fstrim.

root@Ritsu:~# pct fstrim 205
unable to open file '/etc/pve/nodes/Ritsu/lxc/205.conf.tmp.16174' - Input/output      error