Incredible slow vzdump with ayufan differential backup patch - alternatives?

T

taenzerme

Renowned Member
Sep 18, 2013
35
0
71
Bonn, Germany
www.taenzer.de
Hello fellow "proxmoxees",

I'm stuck with a really slow backup of some of our KVM machines.

Enviroment:


  • pve-manager/3.4-6/102d4547 (running kernel: 2.6.32-39-pve)
  • Intel(R) Xeon(R) CPU E5-2620 v2, 64GB RAM
  • NFS storage on Synology RS3614RPxs connected via 10Gbit
  • Differential backup patch from ayufan.eu

pveperf result on nfs share:
Code: 
root@vmhost3:~# pveperf /mnt/pve/storage2
CPU BOGOMIPS:      100548.12
REGEX/SECOND:      1030793
HD SIZE:           7374.94 GB (10.20.30.247:/volume2/vms2)
FSYNCS/SECOND:     2357.50
DNS EXT:           7.76 ms
DNS INT:           14.09 ms (..........)


vms affected config:


  • debian 7
  • virtio disk qcow2 on nfs share, cache=none
  • 150-200gb data inside vm disk, mostly mysql databases and websites (php/html/css files)
  • network: virtio only

example backup log from one vm:

Code: 
501: Jul 31 22:00:02 INFO: Starting Backup of VM 501 (qemu)
501: Jul 31 22:00:02 INFO: status = running
501: Jul 31 22:00:02 INFO: update VM 501: -lock backup
501: Jul 31 22:00:03 INFO: backup mode: snapshot
501: Jul 31 22:00:03 INFO: bandwidth limit: 50000 KB/s
501: Jul 31 22:00:03 INFO: ionice priority: 7
501: Jul 31 22:00:03 INFO: creating archive '/mnt/pve/storage2/dump/vzdump-qemu-501-2015_07_30-22_00_03.vma.lzo--differential-2015_07_31-22_00_02.vcdiff.lzo'
501: Jul 31 22:00:03 INFO: started backup task 'b9753c83-966e-4582-b1f7-2b4397b1d15d'
501: Jul 31 22:00:06 INFO: status: 0% (102170624/429496729600), sparse 0% (33157120), duration 3, 34/23 MB/s
501: Jul 31 22:13:23 INFO: status: 1% (4299030528/429496729600), sparse 0% (104890368), duration 800, 5/5 MB/s
501: Jul 31 22:29:10 INFO: status: 2% (8594259968/429496729600), sparse 0% (178905088), duration 1747, 4/4 MB/s
501: Jul 31 22:31:38 INFO: status: 3% (12897091584/429496729600), sparse 0% (246472704), duration 1895, 29/28 MB/s
501: Jul 31 22:39:39 INFO: status: 4% (17180917760/429496729600), sparse 0% (318775296), duration 2376, 8/8 MB/s
501: Jul 31 22:49:57 INFO: status: 5% (21478768640/429496729600), sparse 0% (418586624), duration 2994, 6/6 MB/s
501: Jul 31 23:02:53 INFO: status: 6% (25771376640/429496729600), sparse 0% (480432128), duration 3770, 5/5 MB/s
501: Jul 31 23:18:17 INFO: status: 7% (30070407168/429496729600), sparse 0% (559599616), duration 4694, 4/4 MB/s
501: Jul 31 23:28:31 INFO: status: 8% (34361835520/429496729600), sparse 0% (643051520), duration 5308, 6/6 MB/s
501: Jul 31 23:35:32 INFO: status: 9% (38663815168/429496729600), sparse 0% (711585792), duration 5729, 10/10 MB/s
501: Jul 31 23:40:29 INFO: status: 10% (42963697664/429496729600), sparse 0% (782962688), duration 6026, 14/14 MB/s
501: Jul 31 23:44:41 INFO: status: 11% (47258927104/429496729600), sparse 0% (848728064), duration 6278, 17/16 MB/s
501: Jul 31 23:53:23 INFO: status: 12% (51542753280/429496729600), sparse 0% (921100288), duration 6800, 8/8 MB/s
501: Jul 31 23:57:59 INFO: status: 13% (55854563328/429496729600), sparse 0% (984903680), duration 7076, 15/15 MB/s
501: Aug 01 00:04:06 INFO: status: 14% (60140814336/429496729600), sparse 0% (1057447936), duration 7443, 11/11 MB/s
501: Aug 01 00:11:56 INFO: status: 15% (64428507136/429496729600), sparse 0% (1136947200), duration 7913, 9/8 MB/s
501: Aug 01 00:21:00 INFO: status: 16% (68726816768/429496729600), sparse 0% (1207209984), duration 8457, 7/7 MB/s
501: Aug 01 00:31:14 INFO: status: 17% (73018900480/429496729600), sparse 0% (1254965248), duration 9071, 6/6 MB/s
501: Aug 01 00:50:38 INFO: status: 18% (77323108352/429496729600), sparse 0% (1299681280), duration 10235, 3/3 MB/s
..... cut ...

501: Aug 01 10:19:34 INFO: status: 46% (197576753152/429496729600), sparse 0% (2959114240), duration 44371, 3/3 MB/s
501: Aug 01 10:41:44 INFO: status: 47% (201864380416/429496729600), sparse 0% (3021684736), duration 45701, 3/3 MB/s
501: Aug 01 10:43:55 ERROR: interrupted by signal
501: Aug 01 10:43:55 INFO: aborting backup job

on the other hand the inital backup of ALL vms on that host at least completed in 11:39:26 for 746.47GB ;-)

Code: 
501: Jul 30 22:00:04 INFO: Starting Backup of VM 501 (qemu)
501: Jul 30 22:00:04 INFO: status = running
501: Jul 30 22:00:04 INFO: update VM 501: -lock backup
501: Jul 30 22:00:05 INFO: backup mode: snapshot
501: Jul 30 22:00:05 INFO: bandwidth limit: 50000 KB/s
501: Jul 30 22:00:05 INFO: ionice priority: 7
501: Jul 30 22:00:05 INFO: creating archive '/mnt/pve/storage2/dump/vzdump-qemu-501-2015_07_30-22_00_03.vma.lzo'
501: Jul 30 22:00:05 INFO: started backup task '30d33d70-b88b-4a25-98df-95bef7f5b766'
501: Jul 30 22:00:08 INFO: status: 0% (154533888/429496729600), sparse 0% (36929536), duration 3, 51/39 MB/s
501: Jul 30 22:01:41 INFO: status: 1% (4332978176/429496729600), sparse 0% (102297600), duration 96, 44/44 MB/s
501: Jul 30 22:03:16 INFO: status: 2% (8637775872/429496729600), sparse 0% (170512384), duration 191, 45/44 MB/s
501: Jul 30 22:04:44 INFO: status: 3% (12923305984/429496729600), sparse 0% (237907968), duration 279, 48/47 MB/s
...

501: Jul 31 00:30:34 INFO: status: 98% (420907253760/429496729600), sparse 3% (15786156032), duration 9029, 49/46 MB/s
501: Jul 31 00:32:00 INFO: status: 99% (425203662848/429496729600), sparse 4% (20001759232), duration 9115, 49/0 MB/s
501: Jul 31 00:33:25 INFO: status: 100% (429496729600/429496729600), sparse 5% (24294825984), duration 9200, 50/0 MB/s
501: Jul 31 00:33:25 INFO: transferred 429496 MB in 9200 seconds (46 MB/s)
501: Jul 31 00:33:25 INFO: archive file size: 184.21GB
501: Jul 31 00:33:26 INFO: Finished Backup of VM 501 (02:33:23)

Any ideas why the differential backup is so incredibly slow?
What are alternatives or configuration recommendations for a reliable backup without storing the full machine data everytime?
 
Last edited:
Open a system monitor like top, or htop while your doing a backup, maybe xdelta clogs your cpu?

It seems like a problem from the patch you applied, maybe ask the author first as this isn't officially supported.
 
Thomas, thanks for answering. I tried getting some help here as the patch does not really have any support forum right now (beside the disqus function on the site). As the normal backup runs fine I guess you're right and I will look into this tonight.

Are there any plans to support differential backups in the future?
 
taenzerme said:
So, basically this already is in QEMU 2.4, which was released some days ago on August 11th ...... and is going to be in Proxmox 8 or so ;-)

Anyway, thanks for the info.
Click to expand...

qemu 2.4 is already in Proxmox VE 4 beta. but the experimental and not feature complete functions like these incremental backups are of course not there.
 
Ah, did not see this was still experimental (aka I just stopped after the "no biggie" part...). Hopefully it will be stable soon as their approach is much better than the one with the patch. Thanks for the clarification.
 
Since a few years I am hoping that this feature is ready and stable, both incrementals as differentials, i guess that in one or two years i will see that these features are ready in PVE.

I am sure that majority of people also want to have these features enabled in PVE, but firstly QEMU has to finish his work.

Moreover, i know that the developers of PVE also want enable a more function when these features are enabled in QEMU: "disaster recovery", and surely in a automatic mode.

Moreover, also i think that with these features enabled, PVE (and maybe other hypervisors based in KVM) will give a big step for enter to large data centers of all world (as in VMware), and for the case of PVE, inclusive without the need of buy the expensive Storages that has High Availability enabled in his setup.

Also i think that the next necessary great step for enter to large data centers of all world will be when "virtio-block" and "virtio-net" has achieved high performance, something like "virtio-blk-dataplane" and "virtio-net-dataplane", and without the loss of the more important features that have QEMU as live migration, the new backup system and disaster recovery (mentioned above), and maybe some other basic functions for large Datacenter that now i don't remember.

Best regards
Cesar
 
mir said:
"virtio-blk-dataplane" and "virtio-net-dataplane" are in the assembly line so expect to see this in qemu-2.5.
Click to expand...

Hi mir, it's a pleasure to greet you again... :-)

Will be fantastic to have such features in QEMU.

Please let me to do a questions:

A- And it with live migration supported?
B- From where did you get such information?

Best regards
Cesar
 
mir said:
Page 28 here: https://videos.cdn.redhat.com/summi...lization-hypervisor-kvm-now-in-the-future.pdf
"RHEL 7.2* RHEV 3.6* IOthreads – virtio-blk data-plane"

RHEL 7.2 will emerge late 2015 or early 2016 which fits nicely with a release of qemu-2.5
Click to expand...

Thanks for the info.
Also, here i have a little more of info:

Official - For calculate the times of wait of each release, You are right mir... :) :
https://access.redhat.com/articles/3078#RHEL7
https://access.redhat.com/support/policy/updates/rhev

Not Official:
http://searchservervirtualization.t...Hat-bares-RHEV-roadmap-fate-of-the-hypervisor
http://www.serverwatch.com/server-news/whats-coming-in-red-hat-enterprise-linux-7.2.html

I do not want to wait for have all these features enabled in PVE !!!!

Best regards
Cesar
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom