Unable to login - I/O Error

[shadowq]

Hi there,

This is the second time I've ran into this issue over the past few weeks. Server working fine, when suddenly I'm unable to log in via Web UI. Syslog shows "Mar 24 21:01:19 s5 pvedaemon[14546]: authentication failure; rhost=xxx user=root@pam msg=error with cfs lock 'authkey': got lock request timeout", plus a bunch of other errors surrounding it:
Mar 24 20:53:05 s5 pvesr[30638]: trying to acquire cfs lock 'file-replication_cfg' ...
Mar 24 20:53:06 s5 pvesr[30638]: trying to acquire cfs lock 'file-replication_cfg' ...
Mar 24 20:53:07 s5 pvesr[30638]: trying to acquire cfs lock 'file-replication_cfg' ...
Mar 24 20:53:08 s5 pvesr[30638]: trying to acquire cfs lock 'file-replication_cfg' ...
Mar 24 20:53:09 s5 pvesr[30638]: error with cfs lock 'file-replication_cfg': got lock request timeout
Mar 24 20:53:09 s5 systemd[1]: pvesr.service: Main process exited, code=exited, status=5/NOTINSTALLED
Mar 24 20:53:09 s5 systemd[1]: pvesr.service: Failed with result 'exit-code'.
Mar 24 20:53:09 s5 systemd[1]: Failed to start Proxmox VE replication runner.
Mar 24 20:53:14 s5 pve-ha-lrm[2061]: unable to write lrm status file - unable to delete old temp file: Input/output error
Mar 24 20:53:19 s5 pve-ha-lrm[2061]: unable to write lrm status file - unable to delete old temp file: Input/output error
Mar 24 20:53:20 s5 pvestatd[1881]: authkey rotation error: error with cfs lock 'authkey': got lock request timeout

Can't figure out what I/O error it's receiving. Partitions are not full. A reboot fixes the issue, but is not sustainable. Any ideas are appreciated!

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-8 (running version: 6.1-8/806edfe1)
pve-kernel-5.3: 6.1-5

 

[Stoiko Ivanov]

seems the cluster filesystem is not running correctly - ist this a cluster?
* check the journal for messages from 'pve-cluster' (cluster filesystem a.k.a pmxcfs) and 'corosync'

I hope this helps!

 

[shadowq]

Thanks for the quick reply, Stoiko. Not a cluster unfortunately. Single node.

The first line in the log that has kicked off the issue both times is
Mar 24 05:40:24 s5 pmxcfs[1780]: [database] crit: commit transaction failed: database or disk is full#010

Which database or disk? No disks are full.

 

[Stoiko Ivanov]

the pmxcfs is a fuse filesystem backed by a sqlite-db (stored in /var/lib/pve-cluster) - which is limited to 30MB in size - check the reference documentation:
https://pve.proxmox.com/pve-docs/chapter-pmxcfs.html

do you have stored large files in /etc/pve?

EDIT: fixed wrong link

 

[shadowq]

Nope!

/dev/fuse 30M 24K 30M 1% /etc/pve

root@s5:/etc/pve# du
2.5K ./priv
5.0K ./nodes
1.0K ./.le
13K .

 

[shadowq]

Turns out following this advice helped:

• On every node do
  systemctl stop pve-cluster
  This may take a while
• On every node do
  sudo rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
• On each node – one by one do
  systemctl start pve-cluster

Which is odd, considering I don't use a cluster? Any ideas as to why this may have happened.

 

[guiambros]

Turns out following this advice helped:

• On every node do
  systemctl stop pve-cluster
  This may take a while
• On every node do
  sudo rm -f /var/lib/pve-cluster/.pmxcfs.lockfile
• On each node – one by one do
  systemctl start pve-cluster

Which is odd, considering I don't use a cluster? Any ideas as to why this may have happened.

Thanks! This also worked for me, and also running a single node; no HA.

In my case I ran out of disk space, and had trouble connecting via the UI. Solved the disk space, but Proxmox was still stuck, and there was a message on syslog every few seconds on pve-ha-lrm[1201]: unable to write lrm status file - unable to open file.

Everything came back to normal after following the steps above.