limit concurrent tasks

[abma]

Is it possible to limit concurrent tasks?

We use a large spinning disk as archive and it becomes really slow when the prune / gc / sync / verify jobs run at the same time.

It would be nice when these tasks can be run sequentially: if a task is already running, wait until its finished and then start the next one.

 

[RolandK]

yes, if i backup a cluster, i see a very large number of concurrent verify tasks when i enable "Verify new backups immediately after completion" option, so many concurrent tasks want to get their share on cpu and Io.

i cannot guess that this is effective with a large number of tasks

 

[abma]

 i cannot guess that this is effective with a large number of tasks

thats why the limit should be configureable. for spinning disks its mostly faster when only one concurrent task is running. The more tasks you start in parallel, the slower it gets because spinning disks are very slow when it goes to random access. Linear access is much faster.

On a spinning disk it would be much more effective having only one task as the bottleneck mostly is the disk.

i guess its best to write a script for that which invokes the tasks via proxmox-backup-manager one by one.

 

[sukerman]

I'm having the same issue, concurrent tasks causes IO delay to hit 70% then its starts crawling

 

[itNGO]

Still not possible in 2.2, but you can create your jobs on PVE that they have enough time to finish and then schedule the next one.
A bit of manual work, but it can work....

 

[k-123]

when our pve nodes expanded from 3 to 5, the IO issue on target pbs goes much worse. (sometimes a small mount of VMs backup will failed)

 

[manwe]

Same issue here. I'd love to see a task queue and max_concurrent_task config option.

 

[Rico29]

+1 !

 

[dignus]

+1

 

[rlcom974]

+1