Intermittent sync issue

ca_maer

Active Member
Dec 5, 2017
181
14
38
42
We have some intermittent sync issue when syncing between 2 PBS.

Here's the log:
Code:
2021-10-31T03:01:27-04:00: sync group vm/108 failed - unable to acquire lock on snapshot directory "/mnt/nas/vm/108/2021-10-30T04:04:17Z" - internal error - tried creating snapshot that's already in use

It's pretty random and is not always the VM 108. There is a PVE sync job to another node that runs around the same time. Could it be the issue ? Is there some sort of workaround for this ?

The PVE sync job is every 15min so it's probably trying to run at 4:00am at the same time as the backup job.

Any ideas ?

Thanks
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,122
1,583
164
could you give the PBS versions on both ends? anything interesting in the logs on either side? what storage(s)/file systems/.. are you using for the datastores?
 

ca_maer

Active Member
Dec 5, 2017
181
14
38
42
Both PBS are in different VMs on different Proxmox servers running ZFS. They both backup on their own NFS share on different NAS servers. Both location are replicated to each other every night. No other errors in the logs other than the one I mentionned in my OP.

Here are the version of both servers:

Code:
()
proxmox-backup: 2.0-1 (running kernel: 5.11.22-5-pve)
proxmox-backup-server: 2.0.9-2 (running version: 2.0.9)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
proxmox-backup-docs: 2.0.9-2
proxmox-backup-client: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-xtermjs: 4.12.0-1
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.5-pve1

Code:
()
proxmox-backup: 2.0-1 (running kernel: 5.11.22-5-pve)
proxmox-backup-server: 2.0.9-2 (running version: 2.0.9)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-4
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
pve-kernel-5.11.22-3-pve: 5.11.22-7
pve-kernel-5.4.124-1-pve: 5.4.124-2
pve-kernel-5.4.106-1-pve: 5.4.106-1
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
proxmox-backup-docs: 2.0.9-2
proxmox-backup-client: 2.0.9-2
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-xtermjs: 4.12.0-1
smartmontools: 7.2-pve2
zfsutils-linux: 2.0.5-pve1
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,122
1,583
164
could you update them to the current version? how are your NFS shares mounted?
 

ca_maer

Active Member
Dec 5, 2017
181
14
38
42
I've updated both to the latest version. The NFS share is mounted using the fstab in the VM
Code:
prd-nas:/mnt/Pool/Backups/PBS /mnt/prd-nas/Backups/PBS nfs defaults 0 0
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,122
1,583
164
and just so I understand correctly, you are syncing from A to B and doing backups on the synced datastore on A, or on B? are those backups combined with a prune-after-backup on the PVE side?
 

ca_maer

Active Member
Dec 5, 2017
181
14
38
42
Not exactly. PBS is running inside different Proxmox PVE VMs. The VMs are replicated using pvesr every 15min while the PBS backup job run daily at 00:00.
Here's a picture to better understand our workflow

pbs-pve-replication.png

Let me know if you need more info !

Thanks
 
Last edited:

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
8,122
1,583
164
okay - so the next time this happens, please check all involved systems for concurrent tasks:
PBS 1 and 2 (anything running on either end in addition to the sync job?)
any PVE or other systems potentially accessing either PBS system (any backups running? any restores? ..)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!