Many 401 Unauthorized messages logged

Menno · Aug 30, 2021

Hi,

We’re seeing quite a few (~10-20 a day) ‘401 Unauthorized’ messaged logged on 4 different clusters but it doesn’t seem to affect services.

The PBS nodes would log messages such as:

GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:172.18.xxx.xxx]:xxxxx] authentication failed - invalid credentials

And the PVE node would log:

backup_vms: error fetching datastores - 401 Unauthorized

It seems to be randomly a different PVE server and each of those servers has the correct permissions and can backup and restore fine.

Acl.cfg:

acl:1:/datastore:pve_hq-test@pbs,pve_hq-test@pbs!backup_pve:DatastoreAudit
acl:1:/datastore/backup_vms:pve_hq-test@pbs,pve_hq-test@pbs!backup_pve:DatastorePowerUser

We’re running an updated PBS (1.1) and PVE (6.4) on all 4 clusters.

proxmox-ve: 6.4-1
proxmox-backup-server: 1.1.13-2

Is anyone else seeing such messages or have an idea how to further debug this? It is only an minor annoyance at this time since everything seems to works just fine.

Thanks.

oguz · Sep 1, 2021

hi,

Menno said:
We’re seeing quite a few (~10-20 a day) ‘401 Unauthorized’ messaged logged on 4 different clusters but it doesn’t seem to affect services.

do you see those attempts all at once, or with some interval inbetween? can you post some logs?

how are the PBS configured in /etc/pve/storage.cfg of the nodes?

Menno said:
We’re running an updated PBS (1.1) and PVE (6.4) on all 4 clusters.

at the moment we're on PBS 2.0 and PVE 7.0, maybe you should upgrade those

Menno · Sep 1, 2021

oguz said:
do you see those attempts all at once, or with some interval inbetween? can you post some logs?

how are the PBS configured in /etc/pve/storage.cfg of the nodes?

It seems pretty random, here are some logs from a few PVE nodes, the backup_1a datastore was only added yesterday and not in use yet.


syslog.13.gz:Aug 19 21:26:25 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.18.gz:Aug 14 08:38:19 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.23.gz:Aug  9 18:53:35 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.23.gz:Aug  9 19:54:25 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.25.gz:Aug  7 11:14:42 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.25.gz:Aug  7 13:34:03 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.28.gz:Aug  4 20:36:08 pve01-dlf pvestatd[4266]: backup_vms: error fetching datastores - 401 Unauthorized

syslog:Sep  1 11:48:48 pve02-dlf pvestatd[3854]: backup_1a: error fetching datastores - 401 Unauthorized
syslog.2.gz:Aug 30 20:43:29 pve01-dlf pvestatd[4266]: backup_1a: error fetching datastores - 401 Unauthorized
syslog.18.gz:Aug 14 20:17:34 pve02-dlf pvestatd[3854]: backup_vms: error fetching datastores - 401 Unauthorized

syslog.1:Aug 31 07:12:52 pve03-dlf pvestatd[4220]: backup_1a: error fetching datastores - 401 Unauthorized
syslog.8.gz:Aug 24 02:34:57 pve03-dlf pvestatd[4220]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.10.gz:Aug 22 10:47:45 pve03-dlf pvestatd[4220]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.30.gz:Aug  2 02:09:29 pve03-dlf pvestatd[4220]: backup_vms: error fetching datastores - 401 Unauthorized

syslog.13.gz:Aug 19 12:18:35 pve04-dlf pvestatd[4074]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.16.gz:Aug 16 11:57:53 pve04-dlf pvestatd[4074]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.27.gz:Aug  5 00:46:50 pve04-dlf pvestatd[4074]: backup_vms: error fetching datastores - 401 Unauthorized

syslog.1:Aug 31 01:47:57 pve05-dlf pvestatd[4607]: backup_1a: error fetching datastores - 401 Unauthorized
syslog.2.gz:Aug 30 05:28:17 pve05-dlf pvestatd[4607]: backup_vms: error fetching datastores - 401 Unauthorized

syslog.8.gz:Aug 24 13:31:07 pve06-dlf pvestatd[5065]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.13.gz:Aug 19 03:55:05 pve06-dlf pvestatd[5065]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.15.gz:Aug 17 15:10:14 pve06-dlf pvestatd[5065]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.26.gz:Aug  6 08:40:27 pve06-dlf pvestatd[5065]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.31.gz:Aug  1 16:39:05 pve06-dlf pvestatd[5065]: backup_vms: error fetching datastores - 401 Unauthorized

syslog.10.gz:Aug 22 01:20:35 pve07-dlf pvestatd[5350]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.13.gz:Aug 19 12:07:43 pve07-dlf pvestatd[5350]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.20.gz:Aug 12 07:54:05 pve07-dlf pvestatd[5350]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.23.gz:Aug  9 22:36:42 pve07-dlf pvestatd[5350]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.27.gz:Aug  5 09:05:07 pve07-dlf pvestatd[5350]: backup_vms: error fetching datastores - 401 Unauthorized
syslog.27.gz:Aug  5 12:33:37 pve07-dlf pvestatd[5350]: backup_vms: error fetching datastores - 401 Unauthorized

2.5 days of logs from the PBS machine shows how often this really happens.


Sep  1 00:16:05 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.153]:48304] authentication failed - invalid credentials
Sep  1 01:46:24 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.102]:57060] authentication failed - invalid credentials
Sep  1 03:07:28 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.205]:46428] authentication failed - invalid credentials
Sep  1 03:36:11 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.154]:55098] authentication failed - invalid credentials
Sep  1 03:36:14 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.153]:37980] authentication failed - invalid credentials
Sep  1 03:46:47 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.152]:35050] authentication failed - invalid credentials
Sep  1 04:30:14 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.104]:34920] authentication failed - invalid credentials
Sep  1 08:06:07 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.101]:56996] authentication failed - invalid credentials
Sep  1 11:48:48 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.202]:45134] authentication failed - invalid credentials
Aug 31 01:47:57 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.205]:56674] authentication failed - invalid credentials
Aug 31 02:21:30 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.151]:60940] authentication failed - invalid credentials
Aug 31 03:00:11 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.102]:53208] authentication failed - invalid credentials
Aug 31 04:10:20 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.104]:57738] authentication failed - invalid credentials
Aug 31 07:12:52 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.203]:59480] authentication failed - invalid credentials
Aug 31 07:44:56 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.202]:42336] authentication failed - invalid credentials
Aug 31 10:43:47 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.101]:43178] authentication failed - invalid credentials
Aug 31 12:32:38 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.202]:39690] authentication failed - invalid credentials
Aug 31 12:33:20 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.103]:47796] authentication failed - invalid credentials
Aug 31 14:53:53 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.102]:34330] authentication failed - invalid credentials
Aug 31 14:55:20 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.154]:53744] authentication failed - invalid credentials
Aug 31 18:05:33 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.102]:54252] authentication failed - invalid credentials
Aug 31 18:42:01 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.155]:44510] authentication failed - invalid credentials
Aug 31 21:06:48 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.204]:40844] authentication failed - invalid credentials
Aug 31 22:05:04 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.102]:58110] authentication failed - invalid credentials
Aug 31 22:43:21 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.155]:49402] authentication failed - invalid credentials
Aug 31 23:04:08 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.205]:45618] authentication failed - invalid credentials
Aug 30 00:31:09 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.104]:55850] authentication failed - invalid credentials
Aug 30 05:28:17 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.205]:38950] authentication failed - invalid credentials
Aug 30 16:00:29 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.201]:46656] authentication failed - invalid credentials
Aug 30 17:11:29 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.106]:45050] authentication failed - invalid credentials
Aug 30 20:43:29 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.201]:48360] authentication failed - invalid credentials
Aug 30 20:55:50 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_1a/status: 401 Unauthorized: [client [::ffff:192.168.1.201]:57266] authentication failed - invalid credentials
Aug 30 21:40:29 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.151]:35030] authentication failed - invalid credentials
Aug 30 21:48:39 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.103]:57658] authentication failed - invalid credentials
Aug 30 22:04:39 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.102]:35536] authentication failed - invalid credentials
Aug 30 22:38:49 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore/backup_vms/status: 401 Unauthorized: [client [::ffff:192.168.1.102]:44110] authentication failed - invalid credentials
Aug 30 23:43:59 pbs01-dlf proxmox-backup-proxy[11265]: GET /api2/json/admin/datastore: 401 Unauthorized: [client [::ffff:192.168.1.102]:60538] authentication failed - invalid credentials

And the relevant part of storage.cfg


pbs: backup_vms
    datastore backup_vms
    server pbs01-dlf.internal.false
    content backup
    fingerprint 50:c5:17:85:2c:fb:91:03:71:4b:a8:29:04:44:5a:a1:d8:f3:f3:a7:15:40:75:a5:56:01:2e:f9:ff:de:fa:6c
    prune-backups keep-all=1
    username pve_dlf@pbs!backup_pve

pbs: backup_1a
    datastore backup_1a
    server pbs01-dlf.internal.false
    content backup
    fingerprint 50:c5:17:85:2c:fb:91:03:71:4b:a8:29:04:44:5a:a1:d8:f3:f3:a7:15:40:75:a5:56:01:2e:f9:ff:de:fa:6c
    prune-backups keep-all=1
    username pve_dlf@pbs!backup_pve

oguz said:
at the moment we're on PBS 2.0 and PVE 7.0, maybe you should upgrade those

I've not had the opportunity to do so yet, I hope to be able to do soon.

oguz · Sep 1, 2021

Menno said:
I've not had the opportunity to do so yet, I hope to be able to do soon.

please try upgrading PBS [0], since there was a similar/related issue which was fixed [1].

if the issue is still there after the upgrade then we can rule out any old bugs.

[0]: https://pbs.proxmox.com/wiki/index.php/Upgrade_from_1.1_to_2.x
[1]: https://lists.proxmox.com/pipermail/pbs-devel/2021-July/003653.html

Menno · Sep 1, 2021

oguz said:
please try upgrading PBS [0], since there was a similar/related issue which was fixed [1].

if the issue is still there after the upgrade then we can rule out any old bugs.

[0]: https://pbs.proxmox.com/wiki/index.php/Upgrade_from_1.1_to_2.x
[1]: https://lists.proxmox.com/pipermail/pbs-devel/2021-July/003653.html

Thanks!

I'll see if I can upgrade one of our machines soon and report back once I have.

oguz · Sep 6, 2021

hi, any news?

Menno · Sep 6, 2021

I need to plan and have some time to do the upgrade, it might take a little while before I'm able to do so.

Search

Search

Many 401 Unauthorized messages logged

Menno

Member

oguz

Proxmox Retired Staff

Menno

Member

oguz

Proxmox Retired Staff

Menno

Member

oguz

Proxmox Retired Staff

Menno

Member