Backup Issue: All nodes keep 90 days except VMs on a single node in cluster?

[devinacosta]

I have a 5-node Proxmox 6.2 cluster, and I noticed today that all my nodes are keeping up to 90 days of backups (of all VMs), except this particular single node in the cluster which is only keeping the last single day?

What is very strange is that they are all backing up to the same storage, which is where you store how many days of backups to keep. Am i missing something that should be obvious?

When I look at the log file i see it clearly mentions it's removing the last backup, but when i look at the storage configuration on that host:

When i manually try to backup a 2nd Backup i get:
INFO: starting new backup job: vzdump 267 --compress zstd --storage synology-sp1 --node admin-virt01 --remove 0 --mode snapshot
ERROR: Backup of VM 267 failed - There is a max backup limit of (1) enforced by the target storage or the vzdump parameters. Either increase the limit or delete old backup(s).
INFO: Failed at 2021-05-25 19:10:42
INFO: Backup job finished with errors
TASK ERROR: job errors

nfs: synology-sp1
export /volume1/storage01
path /mnt/pve/synology-sp1
server 10.X.X.221
content iso,snippets,backup,images,rootdir,vztmpl
maxfiles 90

Totally don't understand why all nodes in the cluster work without issues, except this one node. What would i look at to fix this?

INFO: creating vzdump archive '/mnt/pve/synology-sp1/dump/vzdump-qemu-131-2021_05_25-00_02_22.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '4c5d70b6-0483-4131-a08d-aa17cb71b5e0'
INFO: resuming VM again
INFO: 1% (320.0 MiB of 25.0 GiB) in 3s, read: 106.7 MiB/s, write: 63.3 MiB/s
INFO: 2% (680.0 MiB of 25.0 GiB) in 6s, read: 120.0 MiB/s, write: 67.6 MiB/s
INFO: 4% (1.1 GiB of 25.0 GiB) in 9s, read: 163.6 MiB/s, write: 40.4 MiB/s
INFO: 94% (23.5 GiB of 25.0 GiB) in 2m 27s, read: 210.7 MiB/s, write: 0 B/s
INFO: 96% (24.2 GiB of 25.0 GiB) in 2m 30s, read: 217.3 MiB/s, write: 0 B/s
INFO: 99% (24.8 GiB of 25.0 GiB) in 2m 33s, read: 208.0 MiB/s, write: 0 B/s
INFO: 100% (25.0 GiB of 25.0 GiB) in 2m 35s, read: 122.0 MiB/s, write: 0 B/s
INFO: backup is sparse: 19.85 GiB (79%) total zero data
INFO: transferred 25.00 GiB in 155 seconds (165.2 MiB/s)
INFO: archive file size: 2.19GB
INFO: delete old backup '/mnt/pve/synology-sp1/dump/vzdump-qemu-131-2021_05_24-00_02_17.vma.zst'
INFO: Finished Backup of VM 131 (00:02:35)


One thing i noticed is the nodes that aren't broken are using proxmox-backup-client 0.8.9-1, and the one that is broken is 1.1.5-1. So appears the box that has slightly updated packages is broken. I suspect upgrading all my nodes to the latest version is the fix?

Working nodes:
ii libproxmox-backup-qemu0 0.7.0-1 amd64 Proxmox Backup Server client library for QEMU
ii proxmox-backup-client 0.8.9-1 amd64 Proxmox Backup Client tools

Broken node:

ii libproxmox-backup-qemu0 1.0.3-1 amd64 Proxmox Backup Server client library for QEMU
ii proxmox-backup-client 1.1.5-1 amd64 Proxmox Backup Client tools
ii proxmox-backup-file-restore 1.1.5-3 amd64 Proxmox Backup single file restore tools for pxar and block device backups
ii proxmox-backup-restore-image 0.2.1 amd64 Kernel/initramfs images for Proxmox Backup single-file restore.

 

[fiona]

Hi,
since this is not a PBS storage, the relevant packages would rather be pve-manager and libpve-storage-perl. But yes, it's recommended to use the same package versions for nodes in a cluster.

What is the output of

cat /etc/vzdump.conf
pvesh get /storage/synology-sp1

on the problematic node?

 

[devinacosta]

What is interesting it thinks somehow it has libpve-storage-perl installed from 6.4-1, when it's only running 6.2-12. I presume that something is up with this mismatch somehow?

root@admin-virt01:~# pvesh get /storage/synology-sp1
┌───────────────┬───────────────────────────────────────────┐
│ key │ value │
╞═══════════════╪═══════════════════════════════════════════╡
│ content │ vztmpl,iso,backup,images,rootdir,snippets │
├───────────────┼───────────────────────────────────────────┤
│ digest │ 1131ee4cab8dcc901c8a1e3f13fc940c9da511a6 │
├───────────────┼───────────────────────────────────────────┤
│ export │ /volume1/storage01 │
├───────────────┼───────────────────────────────────────────┤
│ path │ /mnt/pve/synology-sp1 │
├───────────────┼───────────────────────────────────────────┤
│ prune-backups │ keep-last=180 │
├───────────────┼───────────────────────────────────────────┤
│ server │ 10.X.X.221 │
├───────────────┼───────────────────────────────────────────┤
│ shared │ 1 │
├───────────────┼───────────────────────────────────────────┤
│ storage │ synology-sp1 │
├───────────────┼───────────────────────────────────────────┤
│ type │ nfs │
└───────────────┴───────────────────────────────────────────┘
root@admin-virt01:~# cat /etc/vzdump.conf
# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST
#pigz: N
root@admin-virt01:~# dpkg -l | grep libpve-storage-perl
ii libpve-storage-perl 6.4-1 all Proxmox VE storage management library
root@admin-virt01:~# pveversion
pve-manager/6.2-12/b287dd27 (running kernel: 5.4.34-1-pve)

 

[fiona]

What is interesting it thinks somehow it has libpve-storage-perl installed from 6.4-1, when it's only running 6.2-12. I presume that something is up with this mismatch somehow?

Yes, likely the reason.

├───────────────┼───────────────────────────────────────────┤
│ prune-backups │ keep-last=180 │
├───────────────┼───────────────────────────────────────────┤

root@admin-virt01:~# dpkg -l | grep libpve-storage-perl
ii libpve-storage-perl 6.4-1 all Proxmox VE storage management library
root@admin-virt01:~# pveversion
pve-manager/6.2-12/b287dd27 (running kernel: 5.4.34-1-pve)

The old manager does not understand the new prune-backups setting. Please update the node and remember to always use apt-get dist-upgrade or the pveupgrade helper script.