Intermittent sync issue

[ca_maer]

We have some intermittent sync issue when syncing between 2 PBS.

Here's the log:

2021-10-31T03:01:27-04:00: sync group vm/108 failed - unable to acquire lock on snapshot directory "/mnt/nas/vm/108/2021-10-30T04:04:17Z" - internal error - tried creating snapshot that's already in use

It's pretty random and is not always the VM 108. There is a PVE sync job to another node that runs around the same time. Could it be the issue ? Is there some sort of workaround for this ?

The PVE sync job is every 15min so it's probably trying to run at 4:00am at the same time as the backup job.

Any ideas ?

Thanks

 

[ca_maer]

any ideas ?

 

[fabian]

could you give the PBS versions on both ends? anything interesting in the logs on either side? what storage(s)/file systems/.. are you using for the datastores?

 

[ca_maer]

Both PBS are in different VMs on different Proxmox servers running ZFS. They both backup on their own NFS share on different NAS servers. Both location are replicated to each other every night. No other errors in the logs other than the one I mentionned in my OP.

Here are the version of both servers:

()
proxmox-backup: 2.0-1 (running kernel: 5.11.22-5-pve)
proxmox-backup-server: 2.0.9-2 (running version: 2.0.9)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
proxmox-backup-docs: 2.0.9-2
proxmox-backup-client: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-xtermjs: 4.12.0-1
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.5-pve1

()
proxmox-backup: 2.0-1 (running kernel: 5.11.22-5-pve)
proxmox-backup-server: 2.0.9-2 (running version: 2.0.9)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
proxmox-backup-docs: 2.0.9-2
proxmox-backup-client: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-xtermjs: 4.12.0-1
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.5-pve1

 

[fabian]

could you update them to the current version? how are your NFS shares mounted?

 

[ca_maer]

I've updated both to the latest version. The NFS share is mounted using the fstab in the VM

prd-nas:/mnt/Pool/Backups/PBS /mnt/prd-nas/Backups/PBS nfs defaults 0 0

 

[fabian]

and just so I understand correctly, you are syncing from A to B and doing backups on the synced datastore on A, or on B? are those backups combined with a prune-after-backup on the PVE side?

 

[ca_maer]

Not exactly. PBS is running inside different Proxmox PVE VMs. The VMs are replicated using pvesr every 15min while the PBS backup job run daily at 00:00.
Here's a picture to better understand our workflow



Let me know if you need more info !

Thanks

 

[fabian]

okay - so the next time this happens, please check all involved systems for concurrent tasks:
PBS 1 and 2 (anything running on either end in addition to the sync job?)
any PVE or other systems potentially accessing either PBS system (any backups running? any restores? ..)