permission denied - invalid PVE ticket (401)

[mailinglists]

Hi guys,

in PM 6 I got the "permission denied - invalid PVE ticket (401)" when using WEB GUI on one of the cluster nodes.
It logged me out of WEB GUI as soon as I started browsing the effected node via HTTP, even if I originally connected to another node.
Other nodes work fine.

I solved it by restarting services:

root@p37:~# systemctl restart pvedaemon pveproxy

It might be a bug. Seems i'm not the only with with the problem, but I guess writing the solution that worked for me, is the point of this forum post.

Here is the version info

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

 

[Joris L.]

is this not because of an accidental cache clear or session time-out ?

 

[fabian]

usually it's caused by the PVE node having the wrong time..

 

[mailinglists]

My nodes have the same time, they use ntp. This is the first thing I checked.

 

[mailinglists]

is this not because of an accidental cache clear or session time-out ?

No, I did try different browsers, cleared cache, different client PCs, .. etc
It did not work anywhere and started working everywhere after restart of those services on the host.

 

[mailinglists]

Coworker got the same errors, while it worked for me. FYI He was able to fix it after manually removing cookies. No need to restart pve stuff.

 

[mailinglists]

Now even our monitoring system get's kicked out and then just works again.. something strange is happening.. i think we should open a bug...

 

[mailinglists]

As the problems were with only one node, I did systemctl restart pvedaemon pveproxy on that node and no more errors with logging in, not with our monitoring system or with our desktops. This for sure is a bug. Should I open a bug report? @fabian

Workaround would be, to do a cronjob restart of pvedaemon pveproxy every few hours...

 

[hosting]

Got exactly same issue on 1 out of 4 nodes now. Definitely a bug.

proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-4
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-18
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

Time is same/correct on all 4 nodes

 

[fabian]

could you check the logs of pvedaemon and pveproxy for anything suspicious? they should log once a day that the auth key got rotated, if you don't see that message but a warning/error instead then the pmxcfs might have problems..

 

[mailinglists]

Next time I notice logging out, I will grep syslog for pveproxy and pvedaemon.

 

[Joris L.]

Coworker got the same errors, while it worked for me. FYI He was able to fix it after manually removing cookies. No need to restart pve stuff.

There you go

Well, restarting a service does more than just that. It also reinitiates sessions and regenerates relevant ID's. Much same as clearing cookies.

Could it be browser and services have a proxy in between which strips or rewrites part of the header ?

 

[fabian]

There you go

Well, restarting a service does more than just that. It also reinitiates sessions and regenerates relevant ID's. Much same as clearing cookies.

Could it be browser and services have a proxy in between which strips or rewrites part of the header ?

PVE does not have any session state server side..

 

[Joris L.]

PVE does not have any session state server side..

Not even the session-id generated for the cookies ?

 

[fabian]

there is no session-id generated for the cookies. PVE uses a signature based ticket mechanism for the cookies, which allows all nodes in a cluster to verify the tickets without needing per-session state that is synchronized.

 

[Joris L.]

there is no session-id generated for the cookies. PVE uses a signature based ticket mechanism for the cookies, which allows all nodes in a cluster to verify the tickets without needing per-session state that is synchronized.

Thanks for explaining. If i understand correctly this implies the signature is valid across service restarts and no new tickets have to be issues after restart ?

 

[fabian]

yes. the expiration of sessions is entirely time based.

 

[Joris L.]

yes. the expiration of sessions is entirely time based.

now it is clear to me. As an admin i still call such a session albeit not based on a session-ID it does use a form of session identifier (ticket, right ?) Time based or not, as long as connectivity is permitted a session exists, although here purely in an permit window based on what i assume is the ticket stored in the cookie.

 

[T.Herrmann]

Save Solution >>> Login per SSH in every Node and run:

service pve-cluster restart && service pvedaemon restart && service pvestatd restart && service pveproxy restart

 

[mailinglists]

systemctl restart pvedaemon and pveproxy shuld suffice.