permission denied - invalid PVE ticket (401)

Maybe but please test it. My experience with this "Round up" order was good until now.

service pve-cluster restart && service pvedaemon restart && service pvestatd restart && service pveproxy restart

vs.

service pvedaemon restart && service pveproxy restart
 
I think it's not working properly with old servers such as HP G6, G7, and G8. I got this error when I use these servers in the cluster.
Or it's because of the network switch config!
 
Hi, I just encounter this issue in PVE 6.4-15, I have 5 nodes in cluster, found only one node got "permission denied", can not read stat from all the other nodes, I can till login this node directly, but can not read all the other nodes's stata from this node too. So I check the pveproxy service and found it was not working well, so I just restart the pveproxy service on this node and solve this issue:

Code: 
● pveproxy.service - PVE API Proxy Server
   Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2022-11-16 09:34:28 HKT; 4 weeks 1 days ago
  Process: 7025 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
  Process: 7029 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
  Process: 8898 ExecReload=/usr/bin/pveproxy restart (code=exited, status=0/SUCCESS)
 Main PID: 7096 (pveproxy)
    Tasks: 4 (limit: 7372)
   Memory: 213.6M
   CGroup: /system.slice/pveproxy.service
           ├─ 2331 pveproxy worker
           ├─ 7096 pveproxy
           ├─25162 pveproxy worker
           └─40409 pveproxy worker

Dec 15 10:35:14 NODE5 pveproxy[329]: worker exit
Dec 15 10:35:14 NODE5 pveproxy[7096]: worker 329 finished
Dec 15 10:35:14 NODE5 pveproxy[7096]: starting 1 worker(s)
Dec 15 10:35:14 NODE5 pveproxy[7096]: worker 2331 started
Dec 15 10:40:13 NODE5 pveproxy[2331]: Clearing outdated entries from certificate cache
Dec 15 10:45:12 NODE5 pveproxy[25162]: Clearing outdated entries from certificate cache
Dec 15 10:50:13 NODE5 pveproxy[40409]: Clearing outdated entries from certificate cache
Dec 15 11:10:45 NODE5 pveproxy[2331]: Clearing outdated entries from certificate cache
Dec 15 11:18:19 NODE5 pveproxy[25162]: Clearing outdated entries from certificate cache
Dec 15 11:26:44 NODE5 pveproxy[40409]: Clearing outdated entries from certificate cache
Code: 
# pveversion -v
proxmox-ve: 6.4-1 (running kernel: 5.4.203-1-pve)
pve-manager: 6.4-15 (running version: 6.4-15/af7986e6)
pve-kernel-5.4: 6.4-20
pve-kernel-helper: 6.4-20
pve-kernel-5.4.203-1-pve: 5.4.203-1
pve-kernel-5.4.189-2-pve: 5.4.189-2
pve-kernel-5.4.189-1-pve: 5.4.189-1
pve-kernel-5.4.178-1-pve: 5.4.178-1
pve-kernel-5.4.174-2-pve: 5.4.174-2
pve-kernel-5.4.162-1-pve: 5.4.162-2
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.128-1-pve: 5.4.128-2
pve-kernel-5.4.119-1-pve: 5.4.119-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.101-1-pve: 5.4.101-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.78-1-pve: 5.4.78-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.60-1-pve: 5.4.60-2
pve-kernel-5.4.55-1-pve: 5.4.55-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.5-pve2~bpo10+1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve4~bpo10
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.22-pve2~bpo10+1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.1.0-1
libpve-access-control: 6.4-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-5
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.14-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.6-2
pve-cluster: 6.4-1
pve-container: 3.3-6
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-4
pve-firmware: 3.3-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-8
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.7-pve1
P.S, actually, I found still have some issue, so I have to restart the pvedaemon and pvestatd too:
Code: 
systemctl restart pvedaemon pvestatd pveproxy
 
Last edited:
i also get intermittend "Connection error 401: permission denied - invalid PVE ticket" in our 6.4 cluster

typically, this goes away after a while without any action.

how can i debug this ?
 
apparently, the authkey.pub had been changed

the problem went away after doing "touch /etc/pve/authkey.pub"

what was going wrong here ?

i'd be interested why an outdated timestamp on authkey.pub causing intermittend "invalid pve ticket" tickets and logouts in browser.

you can do some action and then another action fails. connecting to console of a VM works and then fails again. reloading browser window fixes it for some action and short after, you get invalid pve ticket again.

why does a simple timestamp update on that file fix such issue?

why does browser behave so weird and non-forseeable?

for me looks a little like buggy behaviour, espectially because it happens out of nowhere. (i have no timesync problem and i did not find any errors in the logs)

Code: 
root@pve-node5:/etc/pve# stat authkey.pub*
  File: authkey.pub
  Size: 451           Blocks: 1          IO Block: 4096   regular file
Device: 41h/65d    Inode: 2858990     Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (   33/www-data)
Access: 2022-12-30 13:42:44.000000000 +0100
Modify: 2022-12-30 13:42:44.000000000 +0100
Change: 2022-12-30 13:42:44.000000000 +0100
 Birth: -
  File: authkey.pub.old
  Size: 451           Blocks: 1          IO Block: 4096   regular file
Device: 41h/65d    Inode: 2858989     Links: 1
Access: (0640/-rw-r-----)  Uid: (    0/    root)   Gid: (   33/www-data)
Access: 2022-12-30 13:42:44.000000000 +0100
Modify: 2022-12-30 13:42:44.000000000 +0100
Change: 2022-12-30 13:42:44.000000000 +0100
 Birth: -
 
root@pve-node5:/etc/pve# date
Fri 30 Dec 2022 02:56:27 PM CET
 
Last edited:
yes , i'm sure it's being caused by the daily authkey rotation. i really suspect a bug or race condition here.

what exactly should i search for in the access log or what journal do you refer to ?

can i force the authkey rotation somehow? i'd like to try to get a repro case
 
RolandK said:
yes , i'm sure it's being caused by the daily authkey rotation. i really suspect a bug or race condition here.
Click to expand...
do you use curl or some other cli tool - or is that in a browser (in a browser simply refreshing once every two hours (actually a bit less) should update the ticket - for curl follow the docs I posted

RolandK said:
what exactly should i search for in the access log or what journal do you refer to ?
Click to expand...
Code: 
Dec 30 10:23:58 rosa pvestatd[13429]: auth key pair too old, rotating..

RolandK said:
can i force the authkey rotation somehow? i'd like to try to get a repro case
Click to expand...
you should be ably to simply touch it with an old timestamp (more than 24 hours) to force rotation - e.g. (for 25 hours from a few seconds ago):
Code: 
touch -d'@1672322814' /etc/pve/authkey.pub*
 
  • Like
Reactions: phauch and RolandK
RolandK said:
what exactly happens when you do a "touch authkey.pub"
Click to expand...
it sets the files mtime (and atime) to now (the ctime gets changed thus as well) - preventing the automated rotation
 
Hello,

So I encountered this same problem and followed this thread after doing touch -d'@1672322814' /etc/pve/authkey.pub*. I am now able to log in to my Proxmox host. Now though, I see all of my VMs with a question mark. Do I need to reboot the PVE host, or what else can I restart without restarting the PVE host?

Thank you.
 

Attachments

  • Screenshot 2023-02-15 at 8.51.02 PM.png
    Screenshot 2023-02-15 at 8.51.02 PM.png
    11.5 KB · Views: 4
jva1601 said:
Hello,

So I encountered this same problem and followed this thread after doing touch -d'@1672322814' /etc/pve/authkey.pub*. I am now able to log in to my Proxmox host. Now though, I see all of my VMs with a question mark. Do I need to reboot the PVE host, or what else can I restart without restarting the PVE host?

Thank you.
Click to expand...

missing status information indicates a problem with pvestatd - check it's status/logs, and restart it if something looks amiss. this can also be caused by a misconfiguration - e.g. a metrics server or storage that is unreachable.
 
Thank you, Fabian! The issue was unreachable storage. After restoring the connection and restarting the host, I can resume normal operations.
 
fabian said:
missing status information indicates a problem with pvestatd - check it's status/logs,
Click to expand...
Where is this log? Is there documentation on pvestatd?

I used the search function in the admin_manual html page, but it "did not match any document".
 
journalctl -b -u pvestatd
 
Seems to be a lot of potential causes for this error.

In my case, this problem was caused by me trying to bring a node with an out-of-date corosync.conf back into the cluster.
Yes my authkey.pub was out of sync, but that was not the cause.

If changes to the cluster configuration are made when a node is offline, the offline node's corosync config is out of date.

you can check if you have this problem by:
grep config_version /etc/corosync/corosync.conf
..and comparing the config version number to the other nodes in the cluster.

If the version on the problematic node is different than the version on the working cluster nodes, then the corosync config is out of date.

If the config versions are out of sync, I fixed it with these steps:
On the out-of-date node with (lower config version number):
systemctl stop pve-cluster
scp /etc/corosync/corosync.conf from a working node to the broken node.
systemctl start pve-cluster

That was enough to fix it for me. Hope that helps.

Cheers,
Smug Llama
 
  • Like
Reactions: Darkk
Hi! ended up in this thread while searching for a solution to mentioned error...
In my case the random "invalid PVE ticket (401)" errors went away after I synchronized the cluster nodes to common NTP time source.
But I still get that error at the end of ISO file upload, the pvestatd log shows "auth key pair too old, rotating..."
However this happens only to large files and while I am at slow link with restricted upload bandwith. No error while uploading smaller ISO images and I could also upload the same large ISO file without error on a fast local LAN.
Do not know the inner logic of this function but me thinks that perhaps during the file upload the session is not refreshed on browser side in the middle of upload operation and the already mentioned key rotation on backend breaks things. And this all comes down to lifespan of key pair...

just my 2 cents to the discussion and perhaps this helps somehow to find a fix. Does anybody know where to configure the key pair lifespan to test the theory?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back