backup usb speed

raqdedicados

Well-Known Member
Nov 26, 2008
85
0
46
Hi there

we re using vzdump in stop mode from the web interface to backup the VM's. the main server is a 2950 with 16gb ram 8 cores and 4 disk sas 15 running at raid 10. attached to this we have a usb disk drive mounted on /backup directory.

As we can see vzump is writting the dump files in about 1mb at a time on the destination directory, this is very slow. so one vm of 4gb demands a lot of time... output of the backup log at the bottom.

(With slow we mean the writing of the /backup/miercoles/ dir the rsync process was pretty fast)

Is there any chance to speed up the proceess. ? why this is so slow ?


Thanks

Diego

pd: we change our backup mode, we were using a samba share but, the samba hangs and suddenly this make all the HW to crash, a kernel panic...strange..so now we will use usb ...

Jul 22 02:39:01 INFO: Starting Backup of VM 101 (openvz)
Jul 22 02:39:01 INFO: status = CTID 101 exist mounted running
Jul 22 02:39:01 INFO: starting first sync /var/lib/vz/private/101 to /var/tmp/vzdumptmp28821
Jul 22 03:01:57 INFO: Number of files: 286638
Jul 22 03:01:57 INFO: Number of files transferred: 266833
Jul 22 03:01:57 INFO: Total file size: 10242706155 bytes
Jul 22 03:01:57 INFO: Total transferred file size: 10057541193 bytes
Jul 22 03:01:57 INFO: Literal data: 10057541266 bytes
Jul 22 03:01:57 INFO: Matched data: 0 bytes
Jul 22 03:01:57 INFO: File list size: 16053367
Jul 22 03:01:57 INFO: File list generation time: 0.001 seconds
Jul 22 03:01:57 INFO: File list transfer time: 0.000 seconds
Jul 22 03:01:57 INFO: Total bytes sent: 10086401551
Jul 22 03:01:57 INFO: Total bytes received: 5354453
Jul 22 03:01:57 INFO: sent 10086401551 bytes received 5354453 bytes 7331460.95 bytes/sec
Jul 22 03:01:57 INFO: total size is 10242706155 speedup is 1.01
Jul 22 03:01:57 INFO: first sync finished (1376 seconds)
Jul 22 03:01:57 INFO: stopping vps
Jul 22 03:01:57 INFO: Stopping container ...
Jul 22 03:02:28 INFO: Container was stopped
Jul 22 03:02:28 INFO: Container is unmounted
Jul 22 03:02:28 INFO: final sync /var/lib/vz/private/101 to /var/tmp/vzdumptmp28821
Jul 22 03:02:39 INFO: Number of files: 286611
Jul 22 03:02:39 INFO: Number of files transferred: 118
Jul 22 03:02:39 INFO: Total file size: 10242759062 bytes
Jul 22 03:02:39 INFO: Total transferred file size: 106103110 bytes
Jul 22 03:02:39 INFO: Literal data: 2938243 bytes
Jul 22 03:02:39 INFO: Matched data: 103164867 bytes
Jul 22 03:02:39 INFO: File list size: 16053311
Jul 22 03:02:39 INFO: File list generation time: 0.001 seconds
Jul 22 03:02:39 INFO: File list transfer time: 0.000 seconds
Jul 22 03:02:39 INFO: Total bytes sent: 19198343
Jul 22 03:02:39 INFO: Total bytes received: 302430
Jul 22 03:02:39 INFO: sent 19198343 bytes received 302430 bytes 1695719.39 bytes/sec
Jul 22 03:02:39 INFO: total size is 10242759062 speedup is 525.25
Jul 22 03:02:39 INFO: final sync finished (11 seconds)
Jul 22 03:02:39 INFO: restarting vps
Jul 22 03:02:39 INFO: Starting container ...
Jul 22 03:02:39 INFO: Container is mounted
Jul 22 03:02:39 INFO: Adding IP address(es): 200.49.152.67
Jul 22 03:02:41 INFO: Setting CPU units: 100000
Jul 22 03:02:41 INFO: Setting CPUs: 1
Jul 22 03:02:41 INFO: Configure meminfo: 1048576
Jul 22 03:02:41 INFO: Set hostname: raq67.raqdedicados.com
Jul 22 03:02:41 INFO: File resolv.conf was modified
Jul 22 03:02:41 INFO: Setting quota ugidlimit: 276447231
Jul 22 03:02:41 INFO: Container start in progress...
Jul 22 03:02:41 INFO: vps is online again after 44 seconds
Jul 22 03:02:41 INFO: creating archive '/backup/miercoles/vzdump-101.dat' (/var/tmp/vzdumptmp28821/101)
Jul 22 03:23:04 INFO: Total bytes written: 10449305600 (9.8GiB, 8.2MiB/s)
Jul 22 03:23:04 INFO: file size 4.87GB
Jul 22 03:23:36 INFO: Finished Backup of VM 101 (00:44:35)
 
  1. what raid controller do you use?
  2. write cache enabled (with BBU)?
  3. Results of 'pveperf' (only run when the server is not under heavy load)
  4. write speed of a normal file to the external USB drive?
 
Well, if you see, it takes double the time it should since it first writes to a temp directory (rsync, then stop the container, then rsync again) (time from 02:39:01 to 03:02:39, total 23 minutes) and at the end compresses (that takes CPU time) and stores in USB (03:02:39 to 03:23:36, total 21 minutes).
See my post about "stop" behaviour, seems a "superstop" could be useful to you too, if, like me, are more concerned about backup time than downtime:
http://www.proxmox.com/forum/showthread.php?t=1668
also:
Total bytes written: 10449305600 (9.8GiB, 8.2MiB/s)
sounds good to me, it transferred 10GB to /var/tmp folder (beware, is in /, so pay attention not to fill it with bigger VM) with 2 rsync, then it compresses (reading 10GB) to a 4GB. I usually have, with less powerful hardware but still with external USB:
...
101: Jul 19 02:57:37 INFO: creating archive '/backup/vm/vzdump-101.dat' (/backup/tempdump/vzdumptmp21607/101)
101: Jul 19 03:51:18 INFO: Total bytes written: 34333675520 (32GiB, 11MiB/s)
101: Jul 19 03:51:18 INFO: file size 20.91GB
101: Jul 19 03:55:39 INFO: Finished Backup of VM 101 (01:55:37)
...
An option could be exclude directory of VPS containing only data (i.e. /serv/smbshare) and backup it with an independent cronjob. Unfortunately exclude dir is not among the options web interface provides (you could enter in a global config, but will work for all vps).
Maybe I'm missing something, I'm not very expert of Proxmox, that anyway appreciate a lot.
 
Tom

We use

perc 6/i with 4 sas drivers 15k running at raid 10

write cache is enabled on the controller

CPU BOGOMIPS: 39904.25
REGEX/SECOND: 642112
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 275.52 MB/sec
AVERAGE SEEK TIME: 15.01 ms
FSYNCS/SECOND: 2123.75

USB: iso file of 700mb would take about 5/6 seconds
thanks!
 
Oh, I should mention that vzdump has a bandwidth limit. The default is "--bwlimit 10240" - max 10MB/s.

Try to set bwlimit in global vzdump config file /etc/vzdump.conf (man vzdump) to a higher value.

- Dietmar
 
to raqdedicados:
" iso file of 700mb would take about 5/6 seconds". Let's suppose it takes 7 seconds -> 100MB/sec -> 800Mbit/s WOW, double the max speed of USB-2 specification (that is 480Mbit/s)!
Sure is not real transfer, maybe if you unmount the device you have to wait until all the data is REALLY wrote to the HD.
 
Hello,

I have the same problem, I think.
Did changing -bwlimit option make it better ?

--
Cordially
Christophe Charron
 
Again, vzdump limits istelfs to 10MB/s. Use -bwlimit option to adjust that.


Where is the file exactly used for? Yesterday I generated the file and entered bwlimit: 20400 because
the backup actually takes>12 hours and I would basicly like to limit backup to the night.

Today it took as long as yesterday. Is /etc/vzdump.conf used as a template? This means
I have to recreate the cronjobs? Or do I have to add the parameter manually into
the cronjobs?

Thanks in advance

Jörn Plewka
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!