multiple backups in parallel

[shantanu]

Hi,

When I schedule a backup of multiple QEMU VMs, they run serially.
Is there any way to make more than one backup run in parallel?
(Knowing that my server is not currently loaded, sometimes I can afford to run 4-5 backups in parallel)

Thanks and Regards,
Shantanu Gadgil

 

[LnxBil]

Hmm .. running one backup does normally saturate at least one of those things: CPU (compression), I/O (reading/writing from/to disk) or network bandwidth (throughput), so running in parallel will not be faster, it'll be slower infact - at least that's what I encountered all the time.

 

[shantanu]

Hi @LxnBill,

during an "in-use" server yes, it could cause negative side effects, but there are times when this could actually help.

The use-case here is when the administrator (IT team) has shutdown all VMs.
Now they want to backup-and-transfer all VMs to another PVE server; possibly an NFS directory which is mounted manually
(this NFS target would typically be the disk on the new server)

The workaround for this is typically to make the new server join the older server in a cluster and migrate one machine at a time.
(which is way slower than the all-in-one approach) and then delete/detach the older server.

This gets trickier if one is migrating from an older version of PVE to a newer one.

Anyway, if the system (server/network/storage) can handle the load, it would be worth doing faster backups.

Regards,
Shantanu

 

[LnxBil]

Anyway, if the system (server/network/storage) can handle the load, it would be worth doing faster backups.

That is exactly my point: the backup process will always saturate on those three things you and I mentioned and therefore running in parallel will not be faster. If you have a "simple" system with one pool of local storage (all VMs on the same physical disks) and one network share to copy to, parallelism will not help (multiple local pools, multiple NICs, multiple LACP ports with multiple network storages on different servers - I'm with you!):

If you cannot read from your local disk fast enough, doing it in parallel will not be faster either. In case of hard disk, it'll slow down significantly
If you saturate your network bandwidth, you will not be faster either.
If your CPU is not able to compress fast enough, use a less demanding compression scheme or no compression at all. In the later case, you're never CPU bound, always disk and network. Normally LZOP compression is transparent in modern CPUs.

The CPU part is the only point I can see where parallelism could be an advantage if you choose to use a demanding compression scheme. It is debatable if you could transfer your VMs faster with higher compression rate in parallel than with a streamlined compression. This reminds of questions like having a trained dog deliver harddisks or upload them via network to your destination :-D

And yes, if you have your big system with 24x NVME and dual or quadruple 100 GBE, your CPU is most certainly your bottleneck with compression, but seriously, with that kind of money, you'd go straight for a real shared storage cluster solution.

The workaround for this is typically to make the new server join the older server in a cluster and migrate one machine at a time.(which is way slower than the all-in-one approach) and then delete/detach the older server.

Yes, slower, but less application downtime - you can even do it live in normal office hours. I never met a customer who would not prefer one-by-one migration during office hours.

(If you have a single server and the potential of migrating it to another system, just go with ZFS ... you have incremental replication and you're really fast migrating VMs - no need for backup&restore on migration with ZFS.)

 

[rootnetworks]

In AMD with NVME modern day , it still run serially may be out of date .
I see in backup processing only use 10% cpu, I have 4x NVME with RAID10; Disk throughput can over 13 GB/s
it's local backup so Nothing is saturated It may be better to provide max Parallel configure for backup task

 

[aeroverra]

To give this question the answer it is actually seeking. Simply delete /run/vzdump.lock and start another backup. Yeah its a bit tedious but still faster than trying to find another way when you only have about 10 vms that are being moved.


May be good to verify its working before deleting your old files but I had no issues. Like the original user I needed stuff off the server that was no longer in use and had 0 bottlenecks. Went from having to potentially wait 20 hours to only waiting about 3. My CPU/ Disk / Network was all below 35% during this process. Kind of a big gap in the current backup system imho.


Results may vary I was on Proxmox 7.2

 

[Tacioandrade]

To give this question the answer it is actually seeking. Simply delete /run/vzdump.lock and start another backup. Yeah its a bit tedious but still faster than trying to find another way when you only have about 10 vms that are being moved.

Thanks for the tip! ;-)

 

[hvisage]

I need to find the more relavent thread on this topic, but the problem I'm facing today, is a specific back takes very long time - and now, it's not bottlenecking on anything other than a single CPU given the nature of the PBS setup/operations over HTTPS on this old hardware.

I know I can run multiple vzdumps, to the PBS from this specific PVE without impacting other stuff, in fact, at the time I need to run them, it'll be beneficial to get it done, instead of postponing the backups over the rest of the next 2 days ;(

 

[homelabenthusiast]

You can use pve-zsync to transfer your vm/lxc to the new server if that's an option. This has very minimal downtime

 

[hvisage]

problem isn't syncing to a new machine, just stating the obvious that I can today, and especially with the new machine, run multiple vzdumps to PBS server(s) without killing the hypervisor, thus the single global lock for vzdump is a tad..... overkill and causing other problems with the sequential nature of vzdumps at present

 

[Tacioandrade]

problem isn't syncing to a new machine, just stating the obvious that I can today, and especially with the new machine, run multiple vzdumps to PBS server(s) without killing the hypervisor, thus the single global lock for vzdump is a tad..... overkill and causing other problems with the sequential nature of vzdumps at present

In this case, what you can do is create a shellscript that checks that if you are running vzdump for this specific vm:
ps -aux | grep vzdump | grep vmid
Delete the file /run/vzdump.lock and place this in the crontab running every 10 minutes for example.

 

[hvisage]

and right there is the problem... hacking and having the ability to have run away processes ;(