[SOLVED] Webui & API logged out every 5 minutes

A

AlmostSerious

Member
Apr 13, 2022
10
2
8
Hello, I am having an issue with the WEBUI logging me out every couple of minutes,
That even happens when the browser is open and actively on the Proxmox VE Web Interface.

I did not have this issues in the beginning. It happend when i was playing around changing the time/date of a LXC Container.
This actually turned out to change also the time/date of my main node.

Now the time is set back to normal, as shown by:

Code: 
timedatectl

               Local time: Thu 2022-05-05 21:02:51 CEST
           Universal time: Thu 2022-05-05 19:02:51 UTC
                 RTC time: Thu 2022-05-05 19:02:51
                Time zone: Europe/Berlin (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no


When im logged out, I usually dont see anything special in the syslog.
Once I saw this reference to the session closed for the user. But that might have just been a coincidence happening at the same time.

Code: 
journalctl -f
May 05 21:17:01 htpc CRON[257770]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
May 05 21:17:01 htpc CRON[257771]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
May 05 21:17:01 htpc CRON[257770]: pam_unix(cron:session): session closed for user root
'
 
Last edited:
Hi, I just tried it on the Edge Browser (same machine) as Im usually on Chrome. Same Behaviour.
I also get the same issue on my Phone.

It's also in a fixed interval. I get the logout in the order of Log-Ins.

i.e.: Opening multiple Sessions:
1. - Edge Browser Machine 1
2. - Chrome Browser Machine 2
3. - Chrome Browser Machine 1

They will log out in the exact same order. Seems to be after a fixed time.
 
So today I realized that also the API is logging me out. So for example when using the HomeAssistant Integration will work fine for the initial login. And then after some minutes will show all containers as unavailable.

This happens every 5minutes & 3 seconds.

1652202921603.png


Output of : journalctl --since -1h -u chrony:

Code: 
-- Journal begins at Sat 2022-02-05 01:00:00 CET, ends at Sun 2024-05-05 02:05:33 CEST. --
Jun 03 02:01:17 htpc chronyd[1128]: Forward time jump detected!
Jun 03 02:01:17 htpc chronyd[1128]: Can't synchronise: no selectable sources
Jun 03 02:04:31 htpc chronyd[1128]: Selected source 193.203.3.170 (2.debian.pool.ntp.org)
Jun 03 02:04:31 htpc chronyd[1128]: System clock wrong by -2703744.493845 seconds
May 05 02:00:28 htpc chronyd[1128]: Forward time jump detected!
May 05 02:00:28 htpc chronyd[1128]: Can't synchronise: no selectable sources
May 05 02:02:39 htpc chronyd[1128]: Selected source 193.203.3.170 (2.debian.pool.ntp.org)
May 05 02:02:39 htpc chronyd[1128]: System clock wrong by -63356160.589662 seconds
-- Boot 9db7ed7b407a48d4ba28ea9143e962af --
May 10 19:31:21 htpc chronyd[1147]: chronyd exiting
May 10 19:31:21 htpc systemd[1]: Stopping chrony, an NTP client/server...
May 10 19:31:22 htpc systemd[1]: chrony.service: Succeeded.
May 10 19:31:22 htpc systemd[1]: Stopped chrony, an NTP client/server.
May 10 19:31:22 htpc systemd[1]: Starting chrony, an NTP client/server...
May 10 19:31:22 htpc chronyd[471774]: chronyd version 4.0 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +ASYN>
May 10 19:31:22 htpc chronyd[471774]: Frequency -17.268 +/- 0.419 ppm read from /var/lib/chrony/chrony.drift
May 10 19:31:22 htpc chronyd[471774]: Using right/UTC timezone to obtain leap second data
May 10 19:31:22 htpc chronyd[471774]: Loaded seccomp filter
May 10 19:31:22 htpc systemd[1]: Started chrony, an NTP client/server.
May 10 19:31:27 htpc chronyd[471774]: Selected source 216.229.4.69 (2.debian.pool.ntp.org)
May 10 19:31:27 htpc chronyd[471774]: System clock TAI offset set to 37 seconds


When it started to happen I had a lot of RRDC Errors and even some Tasks in the Tasklog stuck in the future.
I then deleted /var/lib/rrdcached/db which fixed that.
 
Last edited:
Just to add that I am also experiencing the same issue. I have the Home Assistant integration and 5 minutes after logging in the authentication fails and it is unable to log in from that point onwards. I also have the problem with web UI logging out in the same time period.

I have had problems with RRDC errors and task log events happening in the future. I have cleared the RRDC cache and checked that my system (BIOS) time and the time according to the OS all line up. I am using NTP rather than chrony and that appears to be working correctly as well.
 
bangcrash said:
Just to add that I am also experiencing the same issue. I have the Home Assistant integration and 5 minutes after logging in the authentication fails and it is unable to log in from that point onwards. I also have the problem with web UI logging out in the same time period.

I have had problems with RRDC errors and task log events happening in the future. I have cleared the RRDC cache and checked that my system (BIOS) time and the time according to the OS all line up. I am using NTP rather than chrony and that appears to be working correctly as well.
Click to expand...
It seems to be an issue with API Token refresh.
If I use a Token Generated from within the Proxmox VE WebUI to access some stats I'm never logged out and access continously works.
 
your system seems to jump forward and backward in time - the session tickets contain a time stamp, so if the PVE server jumps in time it might become invalid.

- fix your NTP setup, systems shouldn't jump back and forth - this isn't a time travelling movie after all ;)
- use API tokens for automated API access (they don't require a session login at all, you can just directly make API calls)
 
fabian said:
your system seems to jump forward and backward in time - the session tickets contain a time stamp, so if the PVE server jumps in time it might become invalid.

- fix your NTP setup, systems shouldn't jump back and forth - this isn't a time travelling movie after all ;)
- use API tokens for automated API access (they don't require a session login at all, you can just directly make API calls)
Click to expand...
I don't have a problem with either NTP or system time being mismatched but I am still have the token expiry problems

1652353077797.png
 
you wrote above:

I have had problems with RRDC errors and task log events happening in the future.
Click to expand...

so I'd take that with a grain of salt ;) systemd-timesyncd is also notoriously bad at actually keeping time properly in sync. that being said, there are other reasons where being logged out can happen (authkey being expired and no quorum to rotate for example).
 
fabian said:
you wrote above:



so I'd take that with a grain of salt ;) systemd-timesyncd is also notoriously bad at actually keeping time properly in sync. that being said, there are other reasons where being logged out can happen (authkey being expired and no quorum to rotate for example).
Click to expand...
Is there a way to see the root cause of this in a log file?
The closest I get is by checking the status of the pvedaemon.service where it shows me succesful / unsuccesful auths for the users.
But it doesn't show any details. Ill try with verbose output.

Would you recommend using NTP instead of Chrony?
Is it possible to login to the WebUI using an access token?
 
the log of pvestatd, pvedaemon and pveproxy should contain information about the authkey being too old and being rotated. if that happens twice in a row all tickets signed by the original authkey are invalid, irrespective of time. if you don't have any messages like that in the logs surrounding the time when you are being logged out, you can check the mtime of the authkey files in /etc/pve (the public key part is relevant, it should only be rotated once a day). if your system time-travelled in the past, maybe their timestamps are far in the future?

I'd recommend using chrony for NTP, it's the default for newly installed PVE systems as well. the GUI requires a proper login, an API token is just for automated access (session tickets are treated differently in some parts because of this - protection against CSRF and the like).
 
  • Like
Reactions: bangcrash and AlmostSerious
fabian said:
if your system time-travelled in the past, maybe their timestamps are far in the future?
Click to expand...
Indeed, checking the timestamp shows that my authkey.pub as well as the rotated one both are about 2 years in the future. On the 5th of May. The same time I was able to expereicne this behaviour. Which probably means they stopped rotating at that time.


Code: 
ls -l /etc/pve

-rw-r----- 1 root www-data  451 May  5  2024 authkey.pub
-rw-r----- 1 root www-data  451 May  5  2024 authkey.pub.old
Is it safe to simply delete them? Will they get renewed automatically?

*EDIT*
It worked. Thanks a lot for pointing me into the right direction.
Instead of deleting them, I just edited the files with nano to change the timestamp.
I now stay logged in.
 
Last edited:
  • Like
Reactions: lucasl and bangcrash
AlmostSerious said:
Indeed, checking the timestamp shows that my authkey.pub as well as the rotated one both are about 2 years in the future. On the 5th of May. The same time I was able to expereicne this behaviour. Which probably means they stopped rotating at that time.


Code: 
ls -l /etc/pve

-rw-r----- 1 root www-data  451 May  5  2024 authkey.pub
-rw-r----- 1 root www-data  451 May  5  2024 authkey.pub.old
Is it safe to simply delete them? Will they get renewed automatically?

*EDIT*
It worked. Thanks a lot for pointing me into the right direction.
Instead of deleting them, I just edited the files with nano to change the timestamp.
I now stay logged in.
Click to expand...
I had this problem after moving my Proxmox installation to a new mobo. The above fixed the issue. Thank you.
 
Also fixed for me. I did exactly the same:

touch /etc/pve/authkey.pub*

For those that don't know this simply updates the file timestamp to current time and date
 
You must log in or register to reply here.
Top Bottom