Is Sync During Verify Allowed?

[ericz]

I'm struggling to resolve errors during my sync operation. Below is an example error. While vm 137 is common, I see the same error for other VM IDs.

2024-08-21T06:27:00-07:00: sync group vm/137 failed - unable to acquire lock on snapshot directory "/mnt/datastore/primary/vm/137/2024-08-21T12:00:04Z" - internal error - tried creating snapshot that's already in use

The errors occur mostly, if not always, while the verify operation is active on the same PBS that is running the sync. Is sync during verify allowed?

Here are a few more details on my setup. The intervals are currently set very short to stress the system.

* PVE-WA: Backups sent to PBS-WA every 30 minutes.
* PBS-WA: Verify daily, prune hourly, garbage collect on Saturday.
* PBS-OR: Verify daily, sync to PBS-WA 15 minutes past every hour.

While the start of the verify is deterministic and the start of the sync is deterministic, the end of both is not. Therefore, I'm understanding verifies and syncs as asynchronous operations. As async operations there is a chance that they can be running at the same time. Since that chance exists, intentionally running them at the same time should be okay. If it is not okay how can I guarantee that they will never occur at the same time?

 

[ggoller]

Hi!
no this is not possible, as the sync jobs modifies snapshots and the verify job can't be sure everything was correctly verified at the end. Sadly there is currently no way to schedule jobs, but there is an effort going on to include a scheduler into PBS, which does not only schedule backups, but also other jobs. So you could set the maximum number of parallel jobs to 1 and the verify job will run after the sync job (or vice-versa). (https://bugzilla.proxmox.com/show_bug.cgi?id=3086.) (The bug is about scheduling backups, but will provide the infrastructure to schedule other jobs as well. If this is something you'd really like to see, you can also create another feature request about jobs.)

 

[ericz]

Thank you @ggoller for the definitive answer. I will manually adjust our verify and sync schedules to create some time margin between the two. I do look forward to a PBS update that would be able to manage these conflicts on its own.