Wondering about declining backup-speeds...

Hans Otto Lunde

Active Member
Jul 13, 2017
18
2
43
61
Odder, Denmark
www.egmont-hs.dk
I feel that the speed of backup has declined lately, so I'm wondering, whar other people experience.
My VM's run of a Synology NAS using NFS3 with a 10 gigabit/s link. The NAS itself has it's disks in a RAID6 configuration.
If I test the disk-speed from within one of the Linux-VM's using 'dd' I get something like this for writing

dd if=/dev/zero of=/data/test1.img bs=10G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 5.28242 s, 407 MB/s

and for reads

dd if=/data/test2.img of=/dev/zero bs=10G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1.4855 s, 1.4 GB/s

So reading is MUCH faster which is to be expected from a RAID6 system, I think. Anyway these are the numbers.
When I backup I do it to another share on the same NAS, which will slow things down of course, I would guess about 50%. And then the lzo-compressor will have to do its work too, and there will be overhead etc, etc...
But this is what I get at the moment using lzo-backup on a stopped VM:

INFO: status: 8% (8638824448/10737410), sparse 2% (2342133760), duration 131, read/write 71/42 MB/s
INFO: status: 9% (9709092864/10737410), sparse 2% (2735259648), duration 146, read/write 71/45 MB/s
...
INFO: status: 78% (83767984128/10737410), sparse 47% (51305029632), duration 1123, read/write 74/0 MB/s
INFO: status: 79% (84837662720/10737410), sparse 48% (52374708224), duration 1137, read/write 76/0 MB/s

This is a Windows VM, that I'm backing up. I guess that the backup-system is not file-system aware, but simply backs up the whole diskspace of the VM sector-wise. Is this not the case? I think so, because my backups on windows-systems get quite a big smaller if I wipe the free-space on the VM's filesystem with 0's. You can also tell by the 0 MB/s writes in the second half of the backup-log-entries. Here it's backing up empty diskpace, which compresses to nearly nothing. Correct me if I'm completely wrong here.

But anyway, is this the kind of speed I should expect backing up in this way? I seem to remember, that it used to be a lot faster, but I'm not absolutely sure. Has any of you experienced a slowdown of backups lately?

Hans Otto Lunde
 
Your dd tests are useless. Use fio.
And you forgot, backup is compressed. Thats a big slowdown.
I never saw more than 200MBps for proxmox backup even against local ssd storage.
 
We had some reports the last few weeks about backup speeds dropping but we still haven't identified the cause. Please post the output of 'pveversion -v' and the 'qm config' of all VMs affected.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!