permission denied - invalid PVE ticket (401)

mailinglists

Renowned Member
Mar 14, 2012
641
67
93
Hi guys,

in PM 6 I got the "permission denied - invalid PVE ticket (401)" when using WEB GUI on one of the cluster nodes.
It logged me out of WEB GUI as soon as I started browsing the effected node via HTTP, even if I originally connected to another node.
Other nodes work fine.

I solved it by restarting services:

root@p37:~# systemctl restart pvedaemon pveproxy

It might be a bug. Seems i'm not the only with with the problem, but I guess writing the solution that worked for me, is the point of this forum post.

Here is the version info
Code:
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.0-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2
 
  • Like
Reactions: jebbam
is this not because of an accidental cache clear or session time-out ?
No, I did try different browsers, cleared cache, different client PCs, .. etc
It did not work anywhere and started working everywhere after restart of those services on the host.
 
Coworker got the same errors, while it worked for me. FYI He was able to fix it after manually removing cookies. :) No need to restart pve stuff.
 
  • Like
Reactions: umadu
Now even our monitoring system get's kicked out and then just works again.. something strange is happening.. i think we should open a bug...
1605806513564.png
 

Attachments

  • 1605806414060.png
    1605806414060.png
    60 KB · Views: 40
As the problems were with only one node, I did systemctl restart pvedaemon pveproxy on that node and no more errors with logging in, not with our monitoring system or with our desktops. This for sure is a bug. Should I open a bug report? @fabian

Workaround would be, to do a cronjob restart of pvedaemon pveproxy every few hours...
 
Got exactly same issue on 1 out of 4 nodes now. Definitely a bug.

Code:
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.10-1-pve: 5.3.10-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.4-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-6
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-4
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-18
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

Time is same/correct on all 4 nodes
 
Last edited:
could you check the logs of pvedaemon and pveproxy for anything suspicious? they should log once a day that the auth key got rotated, if you don't see that message but a warning/error instead then the pmxcfs might have problems..
 
Coworker got the same errors, while it worked for me. FYI He was able to fix it after manually removing cookies. :) No need to restart pve stuff.

There you go :)

Well, restarting a service does more than just that. It also reinitiates sessions and regenerates relevant ID's. Much same as clearing cookies.

Could it be browser and services have a proxy in between which strips or rewrites part of the header ?
 
There you go :)

Well, restarting a service does more than just that. It also reinitiates sessions and regenerates relevant ID's. Much same as clearing cookies.

Could it be browser and services have a proxy in between which strips or rewrites part of the header ?

PVE does not have any session state server side..
 
there is no session-id generated for the cookies. PVE uses a signature based ticket mechanism for the cookies, which allows all nodes in a cluster to verify the tickets without needing per-session state that is synchronized.
 
there is no session-id generated for the cookies. PVE uses a signature based ticket mechanism for the cookies, which allows all nodes in a cluster to verify the tickets without needing per-session state that is synchronized.

Thanks for explaining. If i understand correctly this implies the signature is valid across service restarts and no new tickets have to be issues after restart ?
 
yes. the expiration of sessions is entirely time based.
 
  • Like
Reactions: Joris L.
yes. the expiration of sessions is entirely time based.

now it is clear to me. As an admin i still call such a session albeit not based on a session-ID it does use a form of session identifier (ticket, right ?) Time based or not, as long as connectivity is permitted a session exists, although here purely in an permit window based on what i assume is the ticket stored in the cookie.
 
Save Solution >>> Login per SSH in every Node and run:

service pve-cluster restart && service pvedaemon restart && service pvestatd restart && service pveproxy restart
 
  • Like
Reactions: Joris L.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!