Increase backup speed, is this possible?

[fireon]

Hi,

last week we had to recover the hole server. So we observed the the backup/recovery are copy between 40 and 60mb/s, highly variable. So it takes a long time to recover 500GB, back to proxmox. After this we test this with alle formats (lzo, gzip, none). The size of the backuped packages has changend, but not the speed.

Then we this: https://pve.proxmox.com/wiki/Performance_Tweaks#VZdump
and we changed this to one gigabit. Ok, when we to an backup it shows uns the max speed. But by doing the backup the speed is the same.

...but when i copy the image (test.qcow2) direktly with "cp -a" it copies with 117mb/s.

It is possible to change something to get max speed for example on disaster recovery?

Thanks and Regards
fireon

 

[wahmed]

Several things affects backup speed. It is not all about the bandwidth available. When you did copy of .qcow2, it did simple copy without anything else happening thus the 117mb/s speed.

I would say the backup speed depends on the following things:
1. Destination backup storage
2. Online/Offline Backup
3. Backup compression type such as lzo,gz etc.
4. Performance of node the backup is currently running on.
5. kbps limit set in conf.

If you are doing backup to a NFS storage, you will notice that backup speed will jump if you disable async, which is not really recommended to ensure data integrity. You will notice faster backup speed on a fully shutdown VM since it does not have to do lot of things behind the scene to ensure the VM is uninterrupted during backup. To tell you the truth, no matter what you do, you will hit a speed bottleneck in backup speed with the way everything is right now. At least i could not increase the speed significantly. I dealt with the issue by using local SSDs as backup staging storage then copy the backup files manually to another backup storage. My backups of large VMs occurs locally on SSDs, then at later time i move them to backup storage node. You will notice significant performance by doing backup locally on SSD.
This is just one way. I am sure people dealt with many other different ways.

 

[fireon]

Hmm, ok interessting points...
i do some tests with a small vm, and post the results here.

Thanks

 

[fireon]

Here some tests:

vmsize 16GB, realsize 11GB, format qcow2
No limit kb/s

Backup status stop, format --> no compression
Time for Backup 15:0 minutes
Speed: between 15 <--> 26mb/s
Storage: NFS, SYNC, maxspeed on normal copy 117mb/s
Size from *.vma: 10,97GB

Backup status stop, format --> lzo
Time for Backup 4:45 minutes
Speed: 8mb/s 27/mb/s 117mb/s
Storage: NFS, SYNC, maxspeed on normal copy 117mb/s
Size from *.lzo: 3,86GB
note: backup was ready on 56%. I think this depends on used diskspace on VM.
To recover the same VM need 4:20 minutes.


Backup status stop, format --> gzip
Time for Backup 15:00 minutes
Speed: 8mb/s 22/mb/s 45mb/s
Storage: NFS, SYNC, maxspeed on normal copy 117mb/s
Size from *.gz: 2,74GB
note: backup was ready on 56%. I think this depends on used diskspace on VM.

Backup status stop, format --> qcow2
manualcopy "cp -a"
Time for Backup 2:21 minutes
Speed: 66mb/s 82/mb/s 116mb/s
Storage: NFS, SYNC, maxspeed on normal copy 117mb/s
Size from *.qcow2: 11GB
Fastes, but no cron, no interface, always the hole space from VM on backup, no config

Fazit for the first tests: LZO is the best, but in the past some LZO images were damaged on not useable vor recovery. So i'am carefully to use lzo.

 

[fireon]

Backup status stop, format --> gzip (PIGZ 4Cores +1) Symlink on gzip
Time for Backup 06:11 minutes
Speed: 17mb/s 31/mb/s 48mb/s
Storage: NFS, SYNC, maxspeed on normal copy 112mb/s
Size from *.gz: 2,74GB
note: backup was ready on 56%. I think this depends on used diskspace on VM.

So when you use gzip, you should install pigz. I'ts mutch more faster. But be careful. It use's all cores, so backups are only a good idea at time where the services on the server are not used.

To recover the same VM need only 2 minutes.