tl:dr It looks like 3 nodes does not respects set new fingerprint. Configuration shows the good one, but for some reason it shows the old fingerprint in the error output.
I have 6 pve nodes, when I changed certificate, the old fingerprint still stucks somewhere, so I'm getting error "error fetching datastores", but configuration looks ok.
I made some investigation below, but did not nailed it out yet.
Restart problematic server did not help.
Steps to reproduce:
1. remove cert on PBS, so new one will be created.
2. check on PVE nodes, pve1, pve2, pve3 are still holding the very old fingerprint, it's been refreshed on pve4, pve5, pve:
3. I'm manually changing fingerprint from GUI: datacenter >storage >backup1
4.storage.cfg shows that fingerprint is up to date on every node:
5. Manual update on one of problematic nodes does not help:
7. Snippet of configuration
I have 6 pve nodes, when I changed certificate, the old fingerprint still stucks somewhere, so I'm getting error "error fetching datastores", but configuration looks ok.
I made some investigation below, but did not nailed it out yet.
Code:
pbs version: Backup Server 3.1-2
pve version:
➜ ~ ansible pves -m shell -a "pveversion" -i playbooki/inventory
pve1 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve3 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve2 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve4 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)
pve5 | CHANGED | rc=0 >>
pve-manager/8.2.7/3e0176e6bb2ade3b (running kernel: 6.8.12-2-pve)Restart problematic server did not help.
Steps to reproduce:
1. remove cert on PBS, so new one will be created.
New fingerprint is: 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a
2. check on PVE nodes, pve1, pve2, pve3 are still holding the very old fingerprint, it's been refreshed on pve4, pve5, pve:
Code:
$ ansible pves -m shell -a "pvesm list backup1" -i playbooki/inventory[/INDENT]
[INDENT][]pve1 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve2 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve3 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve4 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '95:A8:3BE:A2:7E:26:B3:1B:7A:8B:6F:2A:2F:85:51:95:90:C6:66:8F:E1:B0:C9:BF:33:EA:16:C8:2B1:7A' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '95:A8:3BE:A2:7E:26:B3:1B:7A:8B:6F:2A:2F:85:51:95:90:C6:66:8F:E1:B0:C9:BF:33:EA:16:C8:2B1:7A' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve5 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '95:A8:3BE:A2:7E:26:B3:1B:7A:8B:6F:2A:2F:85:51:95:90:C6:66:8F:E1:B0:C9:BF:33:EA:16:C8:2B1:7A' not verified, abort!non-zero return code3. I'm manually changing fingerprint from GUI: datacenter >storage >backup1
and run the same ansible again:
Code:
$ ansible pves -m shell -a "pvesm list backup1" -i playbooki/inventory[/INDENT]
[INDENT][]pve1 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve2 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return code[/INDENT]
[INDENT][]pve3 | FAILED | rc=255 >>[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!non-zero return codepve, pve4, pve5 shows the right output
4.storage.cfg shows that fingerprint is up to date on every node:
Code:
$ ansible pves -m shell -a "grep finger /etc/pve/storage*" -i playbooki/inventory[/INDENT]
[INDENT][]pve1 | CHANGED | rc=0 >>[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve | CHANGED | rc=0 >>[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve2 | CHANGED | rc=0 >>[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve4 | CHANGED | rc=0 >>[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve3 | CHANGED | rc=0 >>[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][]pve5 | CHANGED | rc=0 >>[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a5. Manual update on one of problematic nodes does not help:
Code:
[root@pve1 ~]# grep finger /etc/pve/storage.cfg[/INDENT]
[INDENT][] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][/INDENT]
[INDENT][][root@pve1 ~]# pvesm set backup1 --fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT][/INDENT]
[INDENT][][root@pve1 ~]# pvesm list backup1[/INDENT]
[INDENT][]backup1: error fetching datastores - fingerprint '60:02:F1:93:B5:54:13:71:05:26:9C0:06:BD:38:44:2D:51:1D:793:2E:E9:57:7E:6E:E1E:75:AE:9F:89' not verified, abort!7. Snippet of configuration
Code:
root@pve1 ~]# cat /etc/pve/storage.cfg[/INDENT]
[INDENT]dir: local[/INDENT]
[INDENT] path /var/lib/vz[/INDENT]
[INDENT] content rootdir,iso,images[/INDENT]
[INDENT] shared 0[/INDENT]
[INDENT][/INDENT]
[INDENT]rbd: cephfs[/INDENT]
[INDENT] content rootdir,images[/INDENT]
[INDENT] krbd 0[/INDENT]
[INDENT] pool cephfs[/INDENT]
[INDENT][/INDENT]
[INDENT]nfs: nas1[/INDENT]
[INDENT] export /volume1/backup[/INDENT]
[INDENT] path /mnt/pve/nas1[/INDENT]
[INDENT] server 192.168.1.230[/INDENT]
[INDENT] content vztmpl,iso,backup[/INDENT]
[INDENT] prune-backups keep-daily=2,keep-hourly=2,keep-monthly=2,keep-weekly=4[/INDENT]
[INDENT][/INDENT]
[INDENT]pbs: backup1[/INDENT]
[INDENT] datastore backup1[/INDENT]
[INDENT] server pbs.dom[/INDENT]
[INDENT] content backup[/INDENT]
[INDENT] fingerprint 95:a8:3b:de:a2:7e:26:b3:1b:7a:8b:6f:2a:2f:85:51:95:90:c6:66:8f:e1:b0:c9:bf:33:ea:16:c8:2b:d1:7a[/INDENT]
[INDENT] prune-backups keep-all=1[/INDENT]
[INDENT] username root@pam
Last edited: