Connection error 401: permission denied - invalid PVE ticket

[immo]

HI,
since a couple of hours I do get this message at the proxmox gui on cluster host 1
but on cluster host 5 everything is fine.

What could be the reason and which steps have to be done to narrow it down

I looked into pveproxy where I guess are the pve tickets from and saw some

Nov 11 13:15:51 prox01 pveproxy[18082]: got inotify poll request in wrong process - disabling inotify

and a few
Nov 11 13:20:39 prox01 pveproxy[8722]: 2020-11-11 13:20:39.497791 +0100 error AnyEvent::Util: Runtime error in AnyEvent::guard callback: Can't call method "_put_session" on an undefined value at /usr/lib/x86_64-linux-gnu/perl5/5.28/AnyEvent/Handle.pm line 2259 during global destruction.

 

[immo]

I dont see them on the working host.

 

[dcsapak]

is the time synced up correctly?

 

[immo]

root@prox05:~# timedatectl timesync-status
Server: ntp1
Poll interval: 34min 8s (min: 32s; max 34min 8s)
Leap: normal
Version: 4
Stratum: 1
Reference: PPS
Precision: 2us (-19)
Root distance: 442us (max: 5s)
Offset: +1.527ms
Delay: 3.383ms
Jitter: 3.488ms
Packet count: 5068
Frequency: +20,113ppm

root@prox01:/var/log# timedatectl timesync-status
Server: ntp1
Poll interval: 34min 8s (min: 32s; max 34min 8s)
Leap: normal
Version: 4
Stratum: 1
Reference: PPS
Precision: 2us (-19)
Root distance: 411us (max: 5s)
Offset: +39us
Delay: 5.539ms
Jitter: 2.028ms
Packet count: 5070
Frequency: +2,812ppm

I guess Yes

 

[dcsapak]

also the time of the client (browser)?

anything else in the syslog?

 

[4ps4all]

I'm getting same problem since today, I can't login through proxmox gui in a single proxmox node (ssh works).
I used lastly pve-promox-backup.iso in another proxmox node, but the two nodes are not in a cluster.
The single node I can't login through proxmox gui should have done vm and ct backups in the other node, I hope so.

pve version:
pve-manager/6.2-15/48bd51b6 (running kernel: 5.4.65-1-pve)

I tried to :
pvecm updatecerts -f
systemctl restart pvedaemon.service pveproxy.service
I tried to reboot
I removed cache from different browsers, I tried different computers, the problem remain the same.

I noticed this in syslog, I dont' know if it is related to the problem:
Nov 18 00:30:06 pve pvedaemon[1006]: <root@pam> successful auth for user 'root@pam'
Nov 18 00:30:10 pve systemd-timesyncd[629]: Synchronized to time server for the first time 37.247.53.178:123 (2.debian.pool.ntp.org).
Nov 18 00:30:22 pve pveproxy[1017]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 256.
Nov 18 00:30:22 pve pveproxy[1015]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 256.
Nov 18 00:30:41 pve systemd[1]: Created slice User Slice of UID 0.
Nov 18 00:30:41 pve systemd[1]: Starting User Runtime Directory /run/user/0...
Nov 18 00:30:41 pve systemd[1]: Started User Runtime Directory /run/user/0.
Nov 18 00:30:41 pve systemd[1]: Starting User Manager for UID 0...
Nov 18 00:30:42 pve systemd[1165]: Listening on GnuPG cryptographic agent and passphrase cache.
Nov 18 00:30:42 pve systemd[1165]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Nov 18 00:30:42 pve systemd[1165]: Listening on GnuPG network certificate management daemon.
Nov 18 00:30:42 pve systemd[1165]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Nov 18 00:30:42 pve systemd[1165]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Nov 18 00:30:42 pve systemd[1165]: Reached target Timers.
Nov 18 00:30:42 pve systemd[1165]: Starting D-Bus User Message Bus Socket.
Nov 18 00:30:42 pve systemd[1165]: Reached target Paths.
Nov 18 00:30:42 pve systemd[1165]: Listening on D-Bus User Message Bus Socket.
Nov 18 00:30:42 pve systemd[1165]: Reached target Sockets.
Nov 18 00:30:42 pve systemd[1165]: Reached target Basic System.
Nov 18 00:30:42 pve systemd[1165]: Reached target Default.
Nov 18 00:30:42 pve systemd[1165]: Startup finished in 311ms.
Nov 18 00:30:42 pve systemd[1]: Started User Manager for UID 0.
Nov 18 00:30:42 pve systemd[1]: Started Session 1 of user root.
Nov 18 00:31:00 pve systemd[1]: Starting Proxmox VE replication runner...
Nov 18 00:31:01 pve systemd[1]: pvesr.service: Succeeded.

what may I try to do?

 

[dcsapak]

what may I try to do?

can you open the browser debug console and see what api calls fail?

 

[mailinglists]

Seems like more people are getting hit by this "issue".
If you can, you can help with debugging as dcsapak asks for, but if you want it just to start working, take a look at my thread, for what to restart.
https://forum.proxmox.com/threads/permission-denied-invalid-pve-ticket-401.78873/#post-349223

 

[4ps4all]

dcsapak said​

can you open the browser debug console and see what api calls fail?

I attached a screenshot of console log of chromium browser.
While blurring the image in some parts the error disappeared.
I had no time to look for a way to "save-the-console-log-in-chrome-to-a-file" when the error disappearead.
I confirm now the proxmox GUI is working. I tried it in Firefox and other PCs in the LAN.
Thanks for your help