Ceph backup speed improvements

[tom]

We got some significant backup speed improvements (virtual disk is stored on Ceph, backup target is a local disk) (pve-qemu-kvm (2.9.0-4) and latest Ceph luminous rc with bluestore).

I reduced backup time in my test VM from 4:21 to 2:17.

Spoiler: backuplog qemu 2.9.0-2

INFO: starting new backup job: vzdump 100 --mode snapshot --storage local --compress lzo --... --node ada1 --remove 0
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: update VM 100: -lock backup
INFO: VM Name: win10
INFO: include disk 'scsi0' 'ceph3:vm-100-disk-1' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-100-2017_08_18-11_10_46.vma.lzo'
INFO: started backup task '0ff12b2b-6f25-4403-91a1-ad3de5cf9aac'
INFO: status: 1% (452984832/34359738368), sparse 0% (158867456), duration 3, read/write 150/98 MB/s
INFO: status: 2% (755040256/34359738368), sparse 0% (205545472), duration 7, read/write 75/63 MB/s
INFO: status: 3% (1047724032/34359738368), sparse 0% (275140608), duration 11, read/write 73/55 MB/s
INFO: status: 4% (1410334720/34359738368), sparse 0% (283160576), duration 17, read/write 60/59 MB/s
INFO: status: 5% (1763704832/34359738368), sparse 0% (286687232), duration 23, read/write 58/58 MB/s
INFO: status: 6% (2094399488/34359738368), sparse 0% (292753408), duration 28, read/write 66/64 MB/s
INFO: status: 7% (2414084096/34359738368), sparse 0% (299159552), duration 33, read/write 63/62 MB/s
.....
INFO: status: 49% (16895967232/34359738368), sparse 9% (3161231360), duration 231, read/write 63/62 MB/s
INFO: status: 50% (17224826880/34359738368), sparse 9% (3163639808), duration 236, read/write 65/65 MB/s
INFO: status: 51% (17605132288/34359738368), sparse 9% (3321286656), duration 240, read/write 95/55 MB/s
INFO: status: 54% (18797101056/34359738368), sparse 12% (4373884928), duration 243, read/write 397/46 MB/s
INFO: status: 67% (23257481216/34359738368), sparse 25% (8834265088), duration 246, read/write 1486/0 MB/s
INFO: status: 80% (27653111808/34359738368), sparse 38% (13229895680), duration 249, read/write 1465/0 MB/s
INFO: status: 93% (32131186688/34359738368), sparse 51% (17707970560), duration 252, read/write 1492/0 MB/s
INFO: status: 100% (34359738368/34359738368), sparse 58% (19936518144), duration 254, read/write 1114/0 MB/s
INFO: transferred 34359 MB in 254 seconds (135 MB/s)
INFO: archive file size: 7.86GB
INFO: Finished Backup of VM 100 (00:04:21)
INFO: Backup job finished successfully
TASK OK

Spoiler: backuplog qemu 2.9.0-4

INFO: starting new backup job: vzdump 100 --storage local --compress lzo --mailto .... --node ada1 --remove 0 --mode snapshot
INFO: Starting Backup of VM 100 (qemu)
INFO: status = running
INFO: update VM 100: -lock backup
INFO: VM Name: win10
INFO: include disk 'scsi0' 'ceph3:vm-100-disk-1' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-100-2017_08_18-11_01_49.vma.lzo'
INFO: started backup task 'dab63e17-a6d4-4186-b2e0-5ffde3c0c65d'
INFO: status: 2% (908460032/34359738368), sparse 0% (223174656), duration 3, read/write 302/228 MB/s
INFO: status: 3% (1334181888/34359738368), sparse 0% (284229632), duration 6, read/write 141/121 MB/s
INFO: status: 5% (1767505920/34359738368), sparse 0% (295653376), duration 9, read/write 144/140 MB/s
INFO: status: 6% (2193227776/34359738368), sparse 0% (305655808), duration 12, read/write 141/138 MB/s
INFO: status: 7% (2580938752/34359738368), sparse 0% (316661760), duration 15, read/write 129/125 MB/s
INFO: status: 8% (2949644288/34359738368), sparse 0% (318869504), duration 18, read/write 122/122 MB/s
INFO: status: 10% (3618635776/34359738368), sparse 1% (583487488), duration 21, read/write 222/134 MB/s
...
INFO: status: 46% (15841886208/34359738368), sparse 9% (3108691968), duration 99, read/write 117/111 MB/s
INFO: status: 47% (16177430528/34359738368), sparse 9% (3110080512), duration 102, read/write 111/111 MB/s
INFO: status: 48% (16500391936/34359738368), sparse 9% (3112677376), duration 105, read/write 107/106 MB/s
INFO: status: 49% (16854024192/34359738368), sparse 9% (3117514752), duration 109, read/write 88/87 MB/s
INFO: status: 50% (17234395136/34359738368), sparse 9% (3120693248), duration 112, read/write 126/125 MB/s
INFO: status: 52% (18022924288/34359738368), sparse 10% (3569455104), duration 115, read/write 262/113 MB/s
INFO: status: 61% (21164457984/34359738368), sparse 19% (6710988800), duration 118, read/write 1047/0 MB/s
INFO: status: 70% (24377294848/34359738368), sparse 28% (9923825664), duration 121, read/write 1070/0 MB/s
INFO: status: 80% (27560771584/34359738368), sparse 38% (13107302400), duration 124, read/write 1061/0 MB/s
INFO: status: 89% (30714888192/34359738368), sparse 47% (16261419008), duration 127, read/write 1051/0 MB/s
INFO: status: 98% (33848033280/34359738368), sparse 56% (19394564096), duration 130, read/write 1044/0 MB/s
INFO: status: 100% (34359738368/34359738368), sparse 57% (19906265088), duration 131, read/write 511/0 MB/s
INFO: transferred 34359 MB in 131 seconds (262 MB/s)
INFO: archive file size: 7.87GB
INFO: Finished Backup of VM 100 (00:02:17)
INFO: Backup job finished successfully
TASK OK

 

[spirit]

Just curious, What is the change exactly ?

 

[dietmar]

We increased the block size when reading data.

 

[spirit]

We increased the block size when reading data.

Wonderfull !

 

[w00dmAn]

Hi there,

How about backup to NFS storage, does it still slow?

Any one can to test it?
I don't have a free test environment

Thanks a lot

Alex

 

[TwiX]

We increased the block size when reading data.

Good, what are the specifications of your test cluster ? osd on ssd or sas ?

Antoine

 

[tom]

Good, what are the specifications of your test cluster ? osd on ssd or sas ?

Antoine

I used the setup and hardware which is described here:
https://www.proxmox.com/en/training/video-tutorials/item/install-ceph-server-on-proxmox-ve

Using no compression on the backup job, I finished in 1:18.

...
INFO: transferred 34359 MB in 75 seconds (458 MB
INFO: archive file size: 13.29GB
INFO: Finished Backup of VM 100 (00:01:18)
INFO: Backup job finished successfully
TASK OK