Unable to login to PVE web gui

[droidus]

I am not able to login to the web gui. I just get a "Login failed. Please try again". I setup authentik (unsuccessfully the other day), but was still able to login with PAM authentication.
Looking in syslog, seeing the following messages:

Jan 19 17:54:33 server pvestatd[2043]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Jan 19 17:54:33 server pvestatd[2043]: status update time (9.969 seconds)

Me attempting to login:

Jan 19 17:54:36 server pvedaemon[1181891]: authentication failure; rhost=::ffff:<ip> user=root@pam msg=cfs-lock 'file-priv_tfa_cfg' error: got lock request timeout

EDIT:
Seeing that systemd-journald is complaining about failing to create new system journal: No space left on device.

 

[Moayad]

hi,

do you have permission to write on /etc/pve/ path?

 

[droidus]

I am logged in as root. Here is what happens when I try to touch a file:

touch: cannot touch 'test': Input/output error

 

[droidus]

Can anyone please assist me?

 

[IxsharpxI]

on a single node i restarted pveproxy and pvedaemon services and that has worked before. or rebooting.

and on clusters ive seen it happen when quorum is not made properly or one machine isnt reporting or something.

 

[droidus]

Did a systemctl restart pveproxy.service pvedaemon.service, and it still fails to login.

EDIT:
Did a status against the two services, and seeing the following:

pvedaemon[198097]: authentication failure; rhost=::ffff:<ip> user=root@pam msg=cfs-lock 'file-priv_tfa_cfg' error: got lock request timeout

pveproxy[198167]: problem with client ::ffff:<ip>; Broken pipe

 

[Moayad]

As I understood you can able in log in using PAM as the root user right?
Do you have a cluster? if yes - are all nodes running?

Can you disable the TFA and try to log in, to disable the TFA from CLI:

grep the ID for the TFA

pvesh get /access/tfa/root@pam

then diable by:

pvesh set /access/tfa/root@pam/29a608f4-b54f-4d4d-ae51-f3e74ef6529c --enable 0

Then try to log in to the PVEWebUI. If that does not help, please do the journalctl -f command during login to PVEWebUI, and post the output

 

[fabian]

Seeing that systemd-journald is complaining about failing to create new system journal: No space left on device.

did you check whether you still have free space? e.g., df -h

 

[droidus]

It seems a reboot fixed it after restarting those services. Not sure why a reboot didn't fix it before.