[SOLVED] Proxmox VE - web gui authentication will fail until a reboot

C

CyberMattSecure

Member
Proxmox Subscriber
Jun 2, 2022
5
1
8
I'm running the latest/up-to-date version of Proxmox VE on a single host currently, randomly the web gui login will refuse to let me log in, giving me the "Login failed. Please try again" error until i reboot the host. I've tried switching the realms but that shouldn't matter as the root login should work on the default realm.

SSH login works just fine for my configured account and the root account.
 
  1. running df -h shows the only disk with usage above 2% is a storage only SSD that is used for a file server, nothing to do with the host or any vm's
  2. Code: 
    root@pve:~# systemctl status pvestatd
● pvestatd.service - PVE Status Daemon
     Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-03-06 10:45:43 EST; 2 days ago
   Main PID: 2865 (pvestatd)
      Tasks: 1 (limit: 154387)
     Memory: 98.1M
        CPU: 1h 46min 32.803s
     CGroup: /system.slice/pvestatd.service
             └─2865 pvestatd

Mar 08 14:49:44 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:49:44 pve pvestatd[2865]: status update time (9.523 seconds)
Mar 08 14:49:53 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:49:53 pve pvestatd[2865]: status update time (9.526 seconds)
Mar 08 14:50:04 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:50:04 pve pvestatd[2865]: status update time (9.516 seconds)
Mar 08 14:50:13 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:50:14 pve pvestatd[2865]: status update time (9.513 seconds)
Mar 08 14:50:23 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:50:23 pve pvestatd[2865]: status update time (9.616 seconds)
  3. Code: 
    root@pve:~# systemctl status pvedaemon
● pvedaemon.service - PVE API Daemon
     Loaded: loaded (/lib/systemd/system/pvedaemon.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-03-06 10:45:44 EST; 2 days ago
   Main PID: 2892 (pvedaemon)
      Tasks: 4 (limit: 154387)
     Memory: 197.2M
        CPU: 20.615s
     CGroup: /system.slice/pvedaemon.service
             ├─2892 pvedaemon
             ├─2893 "pvedaemon worker"
             ├─2894 "pvedaemon worker"
             └─2895 "pvedaemon worker"

Mar 06 13:04:21 pve pvedaemon[2893]: <root@pam> starting task UPID:pve:0000C5CC:000CB564:65E8B025:reloadnetworkall::root@pam:
Mar 06 13:04:22 pve pvesh[50637]: <root@pam> starting task UPID:pve:0000C5D4:000CB5A8:65E8B026:srvreload:networking:root@pam:
Mar 06 13:04:24 pve pvesh[50637]: <root@pam> end task UPID:pve:0000C5D4:000CB5A8:65E8B026:srvreload:networking:root@pam: OK
Mar 06 13:04:24 pve pvedaemon[2893]: <root@pam> end task UPID:pve:0000C5CC:000CB564:65E8B025:reloadnetworkall::root@pam: OK
Mar 06 13:05:42 pve pvedaemon[2894]: <root@pam> update VM 101: -net6 virtio,bridge=isolated,firewall=1
Mar 06 13:14:56 pve pvedaemon[2894]: <root@pam> successful auth for user 'root@pam'
Mar 06 13:45:05 pve pvedaemon[2893]: <root@pam> successful auth for user 'root@pam'
Mar 08 13:39:24 pve pvedaemon[2894]: authentication failure; rhost=::ffff:192.168.5.211 user=root@pam msg=cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:15:48 pve pvedaemon[2895]: authentication failure; rhost=::ffff:192.168.5.211 user=root@pam msg=cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:39:27 pve pvedaemon[2894]: authentication failure; rhost=::ffff:192.168.5.211 user=root@pam msg=cfs-lock 'authkey' error: got lock request timeout
  4. Code: 
    root@pve:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-03-08 14:15:06 EST; 37min ago
    Process: 976881 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=1/FAILURE)
    Process: 976884 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 976885 (pveproxy)
      Tasks: 4 (limit: 154387)
     Memory: 176.2M
        CPU: 1.436s
     CGroup: /system.slice/pveproxy.service
             ├─976885 pveproxy
             ├─976886 "pveproxy worker"
             ├─976887 "pveproxy worker"
             └─976888 "pveproxy worker"

Mar 08 14:15:05 pve systemd[1]: Starting pveproxy.service - PVE API Proxy Server...
Mar 08 14:15:06 pve pvecm[976881]: unable to open file '/etc/pve/priv/authorized_keys.tmp.976883' - Input/output error
Mar 08 14:15:06 pve pveproxy[976884]: Using '/etc/pve/local/pveproxy-ssl.pem' as certificate for the web interface.
Mar 08 14:15:06 pve pveproxy[976885]: starting server
Mar 08 14:15:06 pve pveproxy[976885]: starting 3 worker(s)
Mar 08 14:15:06 pve pveproxy[976885]: worker 976886 started
Mar 08 14:15:06 pve pveproxy[976885]: worker 976887 started
Mar 08 14:15:06 pve pveproxy[976885]: worker 976888 started
Mar 08 14:15:06 pve systemd[1]: Started pveproxy.service - PVE API Proxy Server.
  5. Code: 
    root@pve:~# journalctl -f
Mar 08 14:53:03 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:53:03 pve pvestatd[2865]: status update time (9.530 seconds)
Mar 08 14:53:05 pve pve-ha-lrm[2912]: unable to write lrm status file - unable to open file '/etc/pve/nodes/pve/lrm_status.tmp.2912' - Input/output error
Mar 08 14:53:09 pve pvedaemon[2893]: authentication failure; rhost=::ffff:192.168.5.211 user=root@pam msg=cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:53:10 pve pve-ha-lrm[2912]: unable to write lrm status file - unable to open file '/etc/pve/nodes/pve/lrm_status.tmp.2912' - Input/output error
Mar 08 14:53:11 pve pvescheduler[989448]: jobs: cfs-lock 'file-jobs_cfg' error: got lock request timeout
Mar 08 14:53:11 pve pvescheduler[989447]: replication: cfs-lock 'file-replication_cfg' error: got lock request timeout
Mar 08 14:53:14 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Mar 08 14:53:14 pve pvestatd[2865]: status update time (9.513 seconds)
Mar 08 14:53:15 pve pve-ha-lrm[2912]: unable to write lrm status file - unable to open file '/etc/pve/nodes/pve/lrm_status.tmp.2912' - Input/output error
Mar 08 14:53:20 pve pve-ha-lrm[2912]: unable to write lrm status file - unable to open file '/etc/pve/nodes/pve/lrm_status.tmp.2912' - Input/output error
Mar 08 14:53:23 pve pvestatd[2865]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
 
bbgeek17 said:
Was this a cluster at some point? /etc/pve is a special location that can be using PMXCFS https://pve.proxmox.com/wiki/Proxmox_Cluster_File_System_(pmxcfs)

Clearly you have IO issues to that folder, could be a filesystem corruption, could be bad cluster clean up, could be bad disk.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...

This is a fresh proxmox setup that i did a few weeks back, single instance/node, never was a cluster
 
your webinterface is public avaiable ?
or any other service like ssh ?
 
Last edited:
this is because you get attacks on your host.
i had it too. after reboot the "attack" counter is set to 0 again. wait 5 mins, the attack starts and you cant login anymore.
 
reboot the server and check the logs. there should be seen something like: try root login xxxx times. account blocked
at least i had exactly the issue you describe and if went away after i aftivated the hosters firewall function. before i litterly couldnt sign in with the webinterface (ssh was working)
 
for anyone who might find this thread later the solution was how the backups were configured

something about how it was locking the drives was causing the issue, when i rebooted the system the backups would stop, thus allowing logins again

after disabling my current backup configuration on proxmox the login errors disappeared completely
 
  • Like
Reactions: sdee
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom