[SOLVED] PBS does not respect fingerprint update - "error fetching datastores"

K

kmonti

New Member
Aug 15, 2024
8
1
3
tl:dr It looks like 3 nodes does not respects set new fingerprint. Configuration shows the good one, but for some reason it shows the old fingerprint in the error output.

I have 6 pve nodes, when I changed certificate, the old fingerprint still stucks somewhere, so I'm getting error "error fetching datastores", but configuration looks ok.
I made some investigation below, but did not nailed it out yet.
Code: 
pbs version: Backup Server 3.1-2
pve version:
➜  ~ ansible pves -m shell -a "pveversion" -i playbooki/inventory
pve1 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve3 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve2 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve4 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve5 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)

Restart problematic server did not help.

Steps to reproduce:

1. remove cert on PBS, so new one will be created.
New fingerprint is: 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a​

2. check on PVE nodes, pve1, pve2, pve3 are still holding the very old fingerprint, it's been refreshed on pve4, pve5, pve:
​
Code: 
$ ansible pves -m shell -a "pvesm list backup1" -i playbooki/inventory[/INDENT]
[INDENT][]pve1 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve2 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve3 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve4 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '95:A8:3BE:A2:7E:26:B3:1B:7A:8B:6F:2A:2F:85:51:95:90:C6:66:8F:E1:B0:C9:BF:33:EA:16:C8:2B1:7A' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '95:A8:3BE:A2:7E:26:B3:1B:7A:8B:6F:2A:2F:85:51:95:90:C6:66:8F:E1:B0:C9:BF:33:EA:16:C8:2B1:7A' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve5 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '95:A8:3BE:A2:7E:26:B3:1B:7A:8B:6F:2A:2F:85:51:95:90:C6:66:8F:E1:B0:C9:BF:33:EA:16:C8:2B1:7A' not verified, abort!non-zero return code

3. I'm manually changing fingerprint from GUI: datacenter >storage >backup1

and run the same ansible again:​
Code: 
$ ansible pves -m shell -a "pvesm list backup1" -i playbooki/inventory[/INDENT]
[INDENT][]pve1 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve2 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve3 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code
pve, pve4, pve5 shows the right output​

4.storage.cfg shows that fingerprint is up to date on every node:
Code: 
$ ansible pves -m shell -a "grep finger /etc/pve/storage*" -i playbooki/inventory[/INDENT]
[INDENT][]pve1 | CHANGED | rc=0 >>[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve | CHANGED | rc=0 >>[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve2 | CHANGED | rc=0 >>[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve4 | CHANGED | rc=0 >>[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve3 | CHANGED | rc=0 >>[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve5 | CHANGED | rc=0 >>[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a

5. Manual update on one of problematic nodes does not help:
Code: 
[root@pve1 ~]# grep finger /etc/pve/storage.cfg[/INDENT]
[INDENT][]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][/INDENT]
[INDENT][][root@pve1 ~]# pvesm set backup1 --fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][/INDENT]
[INDENT][][root@pve1 ~]# pvesm list backup1[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!

7. Snippet of configuration
Code: 
root@pve1 ~]# cat /etc/pve/storage.cfg[/INDENT]
[INDENT]dir: local[/INDENT]
[INDENT]    path /var/lib/vz[/INDENT]
[INDENT]    content rootdir,iso,images[/INDENT]
[INDENT]    shared 0[/INDENT]
[INDENT][/INDENT]
[INDENT]rbd: cephfs[/INDENT]
[INDENT]    content rootdir,images[/INDENT]
[INDENT]    krbd 0[/INDENT]
[INDENT]    pool cephfs[/INDENT]
[INDENT][/INDENT]
[INDENT]nfs: nas1[/INDENT]
[INDENT]    export /volume1/backup[/INDENT]
[INDENT]    path /mnt/pve/nas1[/INDENT]
[INDENT]    server 192.168.1.230[/INDENT]
[INDENT]    content vztmpl,iso,backup[/INDENT]
[INDENT]    prune-backups keep-daily=2,keep-hourly=2,keep-monthly=2,keep-weekly=4[/INDENT]
[INDENT][/INDENT]
[INDENT]pbs: backup1[/INDENT]
[INDENT]    datastore backup1[/INDENT]
[INDENT]    server pbs.dom[/INDENT]
[INDENT]    content backup[/INDENT]
[INDENT]    fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT]    prune-backups keep-all=1[/INDENT]
[INDENT]    username root@pam
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back