Backup is slow because too many tasks run at once

[sukerman]

I have a PBS running on NVME but the datastores are on ZFS arrays of HDD's. The problem is scheduled tasks can all run at once so you get in a situation where IO wait is like 30% because the disks are thrashing.

I can't predict how long each task will take so I can't manage this with scheduling.

I have a script that monitors running tasks and when all are complete it shuts down to save energy, then the BIOS on the machine wakes it up at midnight to sync the new backups again, the trouble is they all run at once.

Basicallty I'm look for a way to queue tasks and run one after another to give the HDD's a chance to breath and be more effiecient.

Would a ZIL log dev on an NVME help? if so what size?

Thanks,

 

[Dunuin]

What would help is adding mirrored SSDs as special devices to your HDDs. But primarily for way faster GC, but helps a bit for everything else too, as then no metadata will hit the HDDs so you hit the HDDs IOPS bottleneck a bit later.

There is no orchestrated backup jobs yet. What you could do:
A) limit the bandwidth of each backup job so when all are running in parallel this won't bottleneck the HDDs.
B) follow the manuals recommendation and use local SSDs for your datastore
C) add some orchestration of the backup tasks to your script. You could use the PVE/PBS API to trigger the start of all those backup/verify/prune/GC jobs

PS: Dont forget to check if a scrub/trim/resilver is running and don't shut the PBS down if it does if you don't want to risk losing your backups.

 

[KGundermann]

The problem is scheduled tasks can all run at once so you get in a situation where IO wait is like 30% because the disks are thrashing.

How were you able to run several tasks at once, as PBS has a global lock allowing only one running backup job at a time ??
https://forum.proxmox.com/threads/why-backup-needs-a-global-lock.117477/

 

[VictorSTS]

It's PVE who has the lock so only one backup per PVE host will run. If you have a few nodes in a cluster, each one can run a backup task at once.