Backup bit weired

R

raj

Renowned Member
Sep 17, 2011
219
4
83
www.businessparksolutions.com
Hi team,

I have 1 server and 2 synology DS412+ using as nas .

The 1 nas is an iscsi server where all the images are stored.

The second one has a nfs share where proxmox sends the daily backups.

Both nas have nic teaming and the server has the same.

All running gigabit with a 4006 cisco switch.

Host server is ibm 3650 with 32 GB ram and 2 quad core processor.

Backup is gzip.

I have about 18 vms on that box some windows and other linux.

Now what seems to happen is lets say the backup start at 2100, sometimes the vms backup run smothly and in a few hrs lets say after 10 hrs its done. Other times the job runs for more than 15 hrs. Now looking at the backup logs, it seems to be that if a vm took 20 mins on day one it can take 1 hr on the second day. Its random vms. I have checked usage of the vms and at night they are not used at all.I do not want to use LZO as backup as I had a few misfortunes with that. Does anyone has any ideas why I am getting such an issue?

Cheers,

Raj
 
proxmox-ve-2.6.32: 3.2-129 (running kernel: 2.6.32-30-pve) pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1) pve-kernel-2.6.32-30-pve: 2.6.32-130 pve-kernel-2.6.32-29-pve: 2.6.32-126 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.5-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.5-1 pve-cluster: 3.0-12 qemu-server: 3.1-16 pve-firmware: 1.1-3 libpve-common-perl: 3.0-18 libpve-access-control: 3.0-11 libpve-storage-perl: 3.0-19 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-6 vzctl: 4.0-1pve5 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 1.7-8 ksm-control-daemon: 1.1-1 glusterfs-client: 3.4.2-1
 
Hi Raj,

What type is you nic teaming? Please post your /etc/network/interfaces there.
Just an idea: Can we also confirm that this is not networking ussue. eg. if you have any spare local storage space - just try to backup suspicious vm to that local storage and avoiding use network.

What type of backup you are using? Snapshot, Stop, Suspend? I know one issue with openvz containers. It takes noticeable more time to create archive If there are lots of small files.
 
Hiya,

Here is the config of /etc/network/interfaces


# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
address 192.168.0.11
netmask 255.255.255.0
gateway 192.168.0.251
bridge_ports bond0
bridge_stp off
bridge_fd 0


Here is a df -h

root@Master:/etc/network# df -h
Filesystem Size Used Avail Use% Mounted on
udev 10M 0 10M 0% /dev
tmpfs 3.2G 436K 3.2G 1% /run
/dev/mapper/pve-root 34G 1.2G 31G 4% /
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 6.3G 22M 6.3G 1% /run/shm
/dev/mapper/pve-data 68G 180M 68G 1% /var/lib/vz
/dev/sda2 494M 58M 411M 13% /boot
/dev/fuse 30M 16K 30M 1% /etc/pve
192.168.0.63:/volume1/BackupNight 2.0T 1.5T 489G 76% /mnt/pve/NightlyBackup
192.168.0.63:/volume4/BackupM 985G 511G 475G 52% /mnt/pve/MonthlyBackup
192.168.0.63:/volume5/Archiving 197G 65G 133G 33% /mnt/pve/Archives
192.168.0.62:/volume1/BackupDaily 1.8T 1.3T 596G 68% /mnt/pve/DailyBackup
192.168.0.63:/volume3/ISO 99G 33G 66G 34% /mnt/pve/ISO

Backup type being used is snapshots and all of the vms are kvms.


Hope this help


Cheers,

Raj
 
raj said:
Hiya,
Hope this help
Click to expand...

Thanks for all above.

Just in case please check logs if there are something interesting in your cisco switch. People report on some strange issues with LACP bonding on different types of switches.

You can also monitor how your process is going. Fot little piece of knowledge: I use wonderful "vmstat on steroids" dstat utility. It is great ansi colored tool for real time monitoring. ~$ dstat -maps -N eth0,eth1 60 will show you the utilisation of specified network interfaces as well as CPU, memory etc. I hope this might give you some ideas what's going there.

Cheers.
 
Ok went on the switch and on the int port channel used I could see some drops so rebooted the server and storage and switch and reseated all the ethernet cables then started a new full backup and success. No more drops on the port channel currently.

I backed up using gzip nearly 100GB in 6 hrs.

I keep an eye on to see if there is any changes.

Cheers,

Raj
 
Ok, so after a few days i have the same problem again. The storage is a second ds412+ and both are fine on usage. Here is the log of the backup:

VMID NAME STATUS TIME SIZE FILENAME
111 W OK 00:13:25 2.17GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-111-2014_08_09-21_00_02.vma.gz
117 S OK 00:13:46 3.08GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-117-2014_08_09-21_13_27.vma.gz
118 M OK 00:14:13 2.93GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-118-2014_08_09-21_27_13.vma.gz
119 W OK 01:33:27 1.29GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-119-2014_08_09-21_41_26.vma.gz
200 B1 OK 00:45:19 9.60GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-200-2014_08_09-23_14_53.vma.gz
201 B2 OK 01:24:54 16.89GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-201-2014_08_10-00_00_12.vma.gz
202 B3 OK 01:22:53 21.69GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-202-2014_08_10-01_25_06.vma.gz
203 B4 OK 00:43:48 5.35GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-203-2014_08_10-02_47_59.vma.gz
204 B5 OK 00:40:47 5.78GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-204-2014_08_10-03_31_47.vma.gz
205 B6 OK 03:17:22 12.60GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-205-2014_08_10-04_12_34.vma.gz
206 Z OK 00:38:22 5.38GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-206-2014_08_10-07_29_56.vma.gz
207 B7 OK 00:32:41 6.32GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-207-2014_08_10-08_08_18.vma.gz
210 L OK 00:17:15 3.71GB /mnt/pve/NightlyBackup/dump/vzdump-qemu-210-2014_08_10-08_40_59.vma.gz
TOTAL 11:58:12 96.78GB

Detailed backup logs:

vzdump 111 117 118 119 200 201 202 203 204 205 206 207 210 --quiet 1 --mailto x@x.com --mode snapshot --compress gzip --storage NightlyBackup

111: Aug 09 21:00:02 INFO: Starting Backup of VM 111 (qemu)
111: Aug 09 21:00:02 INFO: status = running
111: Aug 09 21:00:03 INFO: update VM 111: -lock backup
111: Aug 09 21:00:03 INFO: backup mode: snapshot
111: Aug 09 21:00:03 INFO: ionice priority: 7

119: Aug 09 21:41:27 INFO: ionice priority: 7
119: Aug 09 21:41:27 INFO: creating archive '/mnt/pve/NightlyBackup/dump/vzdump-qemu-119-2014_08_09-21_41_26.vma.gz'
119: Aug 09 21:41:27 INFO: started backup task 'be6b443e-c307-4151-a2ff-45c8beba2f2b'
119: Aug 09 21:41:30 INFO: status: 0% (23724032/8606711808), sparse 0% (3039232), duration 3, 7/6 MB/s
119: Aug 09 21:41:35 INFO: status: 1% (114360320/8606711808), sparse 0% (71725056), duration 8, 18/4 MB/s
119: Aug 09 21:41:38 INFO: status: 2% (198311936/8606711808), sparse 1% (149630976), duration 11, 27/2 MB/s
119: Aug 09 21:41:41 INFO: status: 3% (259981312/8606711808), sparse 2% (208498688), duration 14, 20/0 MB/s
119: Aug 09 21:42:07 INFO: status: 4% (347799552/8606711808), sparse 3% (286662656), duration 40, 3/0 MB/s
119: Aug 09 21:42:33 INFO: status: 5% (431816704/8606711808), sparse 3% (332214272), duration 66, 3/1 MB/s
119: Aug 09 21:42:53 INFO: status: 6% (520683520/8606711808), sparse 3% (337584128), duration 86, 4/4 MB/s
119: Aug 09 21:43:12 INFO: status: 7% (604372992/8606711808), sparse 4% (346025984), duration 105, 4/3 MB/s
119: Aug 09 21:43:49 INFO: status: 8% (688848896/8606711808), sparse 4% (356814848), duration 142, 2/1 MB/s
119: Aug 09 21:44:53 INFO: status: 9% (775487488/8606711808), sparse 4% (357974016), duration 206, 1/1 MB/s
119: Aug 09 21:45:28 INFO: status: 10% (862912512/8606711808), sparse 4% (364101632), duration 241, 2/2 MB/s
119: Aug 09 21:46:00 INFO: status: 11% (949616640/8606711808), sparse 4% (380297216), duration 273, 2/2 MB/s
119: Aug 09 21:46:35 INFO: status: 12% (1032847360/8606711808), sparse 4% (399437824), duration 308, 2/1 MB/s
119: Aug 09 21:47:24 INFO: status: 13% (1119813632/8606711808), sparse 4% (410591232), duration 357, 1/1 MB/s
119: Aug 09 21:48:23 INFO: status: 14% (1205338112/8606711808), sparse 5% (434020352), duration 416, 1/1 MB/s
119: Aug 09 21:49:15 INFO: status: 15% (1291124736/8606711808), sparse 5% (435920896), duration 468, 1/1 MB/s
119: Aug 09 21:50:20 INFO: status: 16% (1377566720/8606711808), sparse 5% (436211712), duration 533, 1/1 MB/s
119: Aug 09 21:51:25 INFO: status: 17% (1463615488/8606711808), sparse 5% (436895744), duration 598, 1/1 MB/s
119: Aug 09 21:53:01 INFO: status: 18% (1550516224/8606711808), sparse 5% (439287808), duration 694, 0/0 MB/s
119: Aug 09 21:54:14 INFO: status: 19% (1635516416/8606711808), sparse 5% (441786368), duration 767, 1/1 MB/s
119: Aug 09 21:55:48 INFO: status: 20% (1721565184/8606711808), sparse 5% (443928576), duration 861, 0/0 MB/s
119: Aug 09 21:57:17 INFO: status: 21% (1807613952/8606711808), sparse 5% (449421312), duration 950, 0/0 MB/s
119: Aug 09 21:58:28 INFO: status: 22% (1893859328/8606711808), sparse 5% (451575808), duration 1021, 1/1 MB/s
119: Aug 09 21:59:38 INFO: status: 23% (1979645952/8606711808), sparse 5% (451948544), duration 1091, 1/1 MB/s
119: Aug 09 22:01:48 INFO: status: 24% (2066546688/8606711808), sparse 5% (469749760), duration 1221, 0/0 MB/s
119: Aug 09 22:03:01 INFO: status: 25% (2153971712/8606711808), sparse 6% (522477568), duration 1294, 1/0 MB/s
119: Aug 09 22:04:17 INFO: status: 26% (2237792256/8606711808), sparse 6% (525717504), duration 1370, 1/1 MB/s
119: Aug 09 22:05:56 INFO: status: 27% (2324103168/8606711808), sparse 6% (567480320), duration 1469, 0/0 MB/s
119: Aug 09 22:07:26 INFO: status: 28% (2410151936/8606711808), sparse 7% (623079424), duration 1559, 0/0 MB/s
119: Aug 09 22:08:48 INFO: status: 29% (2496397312/8606711808), sparse 8% (698638336), duration 1641, 1/0 MB/s
119: Aug 09 22:10:02 INFO: status: 30% (2582380544/8606711808), sparse 8% (741498880), duration 1715, 1/0 MB/s
119: Aug 09 22:11:50 INFO: status: 31% (2668232704/8606711808), sparse 8% (741498880), duration 1823, 0/0 MB/s
119: Aug 09 22:13:14 INFO: status: 32% (2754215936/8606711808), sparse 8% (741629952), duration 1907, 1/1 MB/s
119: Aug 09 22:14:24 INFO: status: 33% (2840330240/8606711808), sparse 8% (743370752), duration 1977, 1/1 MB/s
119: Aug 09 22:15:45 INFO: status: 34% (2926444544/8606711808), sparse 8% (753844224), duration 2058, 1/0 MB/s
119: Aug 09 22:17:43 INFO: status: 35% (3012952064/8606711808), sparse 8% (767758336), duration 2176, 0/0 MB/s
119: Aug 09 22:18:49 INFO: status: 36% (3098542080/8606711808), sparse 9% (778313728), duration 2242, 1/1 MB/s
119: Aug 09 22:19:54 INFO: status: 37% (3185704960/8606711808), sparse 9% (782487552), duration 2307, 1/1 MB/s
119: Aug 09 22:21:51 INFO: status: 38% (3271360512/8606711808), sparse 9% (791695360), duration 2424, 0/0 MB/s
119: Aug 09 22:23:14 INFO: status: 39% (3357802496/8606711808), sparse 9% (825982976), duration 2507, 1/0 MB/s
119: Aug 09 22:24:25 INFO: status: 40% (3443064832/8606711808), sparse 9% (834301952), duration 2578, 1/1 MB/s
119: Aug 09 22:25:31 INFO: status: 41% (3529703424/8606711808), sparse 10% (875057152), duration 2644, 1/0 MB/s
119: Aug 09 22:26:48 INFO: status: 42% (3616079872/8606711808), sparse 10% (912683008), duration 2721, 1/0 MB/s
119: Aug 09 22:27:45 INFO: status: 43% (3701342208/8606711808), sparse 10% (918106112), duration 2778, 1/1 MB/s
119: Aug 09 22:28:39 INFO: status: 44% (3787456512/8606711808), sparse 11% (948670464), duration 2832, 1/1 MB/s
119: Aug 09 22:29:41 INFO: status: 45% (3875078144/8606711808), sparse 11% (977203200), duration 2894, 1/0 MB/s
119: Aug 09 22:31:16 INFO: status: 46% (3959160832/8606711808), sparse 11% (985923584), duration 2989, 0/0 MB/s
119: Aug 09 22:32:38 INFO: status: 47% (4045537280/8606711808), sparse 11% (1001844736), duration 3071, 1/0 MB/s
119: Aug 09 22:33:47 INFO: status: 48% (4132569088/8606711808), sparse 11% (1020067840), duration 3140, 1/0 MB/s
119: Aug 09 22:35:05 INFO: status: 49% (4217831424/8606711808), sparse 12% (1037398016), duration 3218, 1/0 MB/s
119: Aug 09 22:36:41 INFO: status: 50% (4304470016/8606711808), sparse 12% (1075691520), duration 3314, 0/0 MB/s
119: Aug 09 22:37:56 INFO: status: 51% (4390256640/8606711808), sparse 13% (1161478144), duration 3389, 1/0 MB/s
119: Aug 09 22:39:06 INFO: status: 52% (4476108800/8606711808), sparse 14% (1247330304), duration 3459, 1/0 MB/s
119: Aug 09 22:40:26 INFO: status: 53% (4562354176/8606711808), sparse 15% (1328574464), duration 3539, 1/0 MB/s
119: Aug 09 22:41:52 INFO: status: 54% (4648337408/8606711808), sparse 16% (1407410176), duration 3625, 0/0 MB/s
119: Aug 09 22:43:15 INFO: status: 55% (4735172608/8606711808), sparse 16% (1445404672), duration 3708, 1/0 MB/s
119: Aug 09 22:44:32 INFO: status: 56% (4820566016/8606711808), sparse 16% (1447149568), duration 3785, 1/1 MB/s
119: Aug 09 22:46:05 INFO: status: 57% (4905828352/8606711808), sparse 16% (1457623040), duration 3878, 0/0 MB/s
119: Aug 09 22:47:19 INFO: status: 58% (4992139264/8606711808), sparse 17% (1543933952), duration 3952, 1/0 MB/s
119: Aug 09 22:48:28 INFO: status: 59% (5079040000/8606711808), sparse 18% (1630834688), duration 4021, 1/0 MB/s
119: Aug 09 22:49:29 INFO: status: 60% (5164040192/8606711808), sparse 19% (1715834880), duration 4082, 1/0 MB/s
119: Aug 09 22:51:15 INFO: status: 61% (5251727360/8606711808), sparse 20% (1803522048), duration 4188, 0/0 MB/s
119: Aug 09 22:52:31 INFO: status: 62% (5336596480/8606711808), sparse 21% (1888391168), duration 4264, 1/0 MB/s
119: Aug 09 22:53:38 INFO: status: 63% (5423431680/8606711808), sparse 22% (1975226368), duration 4331, 1/0 MB/s
119: Aug 09 22:54:38 INFO: status: 64% (5508431872/8606711808), sparse 23% (2060226560), duration 4391, 1/0 MB/s
119: Aug 09 22:56:06 INFO: status: 65% (5595136000/8606711808), sparse 24% (2146930688), duration 4479, 0/0 MB/s
119: Aug 09 22:57:16 INFO: status: 66% (5680594944/8606711808), sparse 25% (2232389632), duration 4549, 1/0 MB/s
119: Aug 09 22:58:36 INFO: status: 67% (5767430144/8606711808), sparse 26% (2319224832), duration 4629, 1/0 MB/s
119: Aug 09 22:59:43 INFO: status: 68% (5852626944/8606711808), sparse 27% (2404421632), duration 4696, 1/0 MB/s
119: Aug 09 23:01:53 INFO: status: 69% (5939396608/8606711808), sparse 28% (2491191296), duration 4826, 0/0 MB/s
119: Aug 09 23:03:03 INFO: status: 70% (6025641984/8606711808), sparse 29% (2577436672), duration 4896, 1/0 MB/s
119: Aug 09 23:03:42 INFO: status: 71% (6110904320/8606711808), sparse 30% (2662699008), duration 4935, 2/0 MB/s
119: Aug 09 23:04:16 INFO: status: 72% (6198001664/8606711808), sparse 31% (2749796352), duration 4969, 2/0 MB/s
119: Aug 09 23:04:50 INFO: status: 73% (6284443648/8606711808), sparse 32% (2836238336), duration 5003, 2/0 MB/s
119: Aug 09 23:05:19 INFO: status: 74% (6370623488/8606711808), sparse 33% (2922418176), duration 5032, 2/0 MB/s
119: Aug 09 23:05:58 INFO: status: 75% (6459686912/8606711808), sparse 34% (3011481600), duration 5071, 2/0 MB/s
119: Aug 09 23:06:22 INFO: status: 76% (6542852096/8606711808), sparse 35% (3094646784), duration 5095, 3/0 MB/s
119: Aug 09 23:06:46 INFO: status: 77% (6628769792/8606711808), sparse 36% (3180564480), duration 5119, 3/0 MB/s
119: Aug 09 23:07:18 INFO: status: 78% (6714294272/8606711808), sparse 37% (3264782336), duration 5151, 2/0 MB/s
119: Aug 09 23:08:02 INFO: status: 79% (6802440192/8606711808), sparse 38% (3349114880), duration 5195, 2/0 MB/s
119: Aug 09 23:08:41 INFO: status: 80% (6887440384/8606711808), sparse 39% (3400126464), duration 5234, 2/0 MB/s
119: Aug 09 23:09:17 INFO: status: 81% (6973358080/8606711808), sparse 40% (3486044160), duration 5270, 2/0 MB/s
119: Aug 09 23:09:55 INFO: status: 82% (7058685952/8606711808), sparse 41% (3571372032), duration 5308, 2/0 MB/s
119: Aug 09 23:10:32 INFO: status: 83% (7146373120/8606711808), sparse 42% (3659059200), duration 5345, 2/0 MB/s
119: Aug 09 23:11:11 INFO: status: 84% (7231111168/8606711808), sparse 43% (3743797248), duration 5384, 2/0 MB/s
119: Aug 09 23:11:45 INFO: status: 85% (7318798336/8606711808), sparse 44% (3831484416), duration 5418, 2/0 MB/s
119: Aug 09 23:12:23 INFO: status: 86% (7403798528/8606711808), sparse 45% (3916484608), duration 5456, 2/0 MB/s
119: Aug 09 23:12:57 INFO: status: 87% (7488208896/8606711808), sparse 46% (4000894976), duration 5490, 2/0 MB/s
119: Aug 09 23:13:28 INFO: status: 88% (7574847488/8606711808), sparse 47% (4087533568), duration 5521, 2/0 MB/s
119: Aug 09 23:14:05 INFO: status: 89% (7660765184/8606711808), sparse 48% (4173451264), duration 5558, 2/0 MB/s
119: Aug 09 23:14:16 INFO: status: 90% (7746879488/8606711808), sparse 49% (4259565568), duration 5569, 7/0 MB/s
119: Aug 09 23:14:20 INFO: status: 91% (7833583616/8606711808), sparse 50% (4346269696), duration 5573, 21/0 MB/s
119: Aug 09 23:14:25 INFO: status: 92% (7937654784/8606711808), sparse 51% (4450340864), duration 5578, 20/0 MB/s
119: Aug 09 23:14:28 INFO: status: 93% (8016494592/8606711808), sparse 52% (4529180672), duration 5581, 26/0 MB/s
119: Aug 09 23:14:31 INFO: status: 94% (8095793152/8606711808), sparse 53% (4603236352), duration 5584, 26/1 MB/s
119: Aug 09 23:14:35 INFO: status: 95% (8182497280/8606711808), sparse 54% (4689940480), duration 5588, 21/0 MB/s
119: Aug 09 23:14:38 INFO: status: 96% (8263958528/8606711808), sparse 55% (4771401728), duration 5591, 27/0 MB/s
119: Aug 09 23:14:42 INFO: status: 97% (8369930240/8606711808), sparse 56% (4877373440), duration 5595, 26/0 MB/s
119: Aug 09 23:14:46 INFO: status: 98% (8446738432/8606711808), sparse 57% (4954181632), duration 5599, 19/0 MB/s
119: Aug 09 23:14:49 INFO: status: 99% (8533114880/8606711808), sparse 58% (5040558080), duration 5602, 28/0 MB/s
119: Aug 09 23:14:53 INFO: status: 100% (8606711808/8606711808), sparse 59% (5109960704), duration 5606, 18/1 MB/s
119: Aug 09 23:14:53 INFO: transferred 8606 MB in 5606 seconds (1 MB/s)
119: Aug 09 23:14:53 INFO: archive file size: 1.29GB
119: Aug 09 23:14:53 INFO: delete old backup '/mnt/pve/NightlyBackup/dump/vzdump-qemu-119-2014_08_02-22_39_23.vma.gz'
119: Aug 09 23:14:53 INFO: Finished Backup of VM 119 (01:33:27)


201: Aug 10 00:00:12 INFO: Starting Backup of VM 201 (qemu)
201: Aug 10 00:00:12 INFO: status = running
201: Aug 10 00:00:13 INFO: update VM 201: -lock backup
201: Aug 10 00:00:13 INFO: backup mode: snapshot
201: Aug 10 00:00:13 INFO: ionice priority: 7
201: Aug 10 00:00:13 INFO: creating archive '/mnt/pve/NightlyBackup/dump/vzdump-qemu-201-2014_08_10-00_00_12.vma.gz'
201: Aug 10 00:00:13 INFO: started backup task '5292ce1f-d1a0-4cc5-92c8-9a6c4315dcf0'
201: Aug 10 00:00:16 INFO: status: 0% (23003136/42949672960), sparse 0% (7720960), duration 3, 7/5 MB/s
201: Aug 10 00:01:06 INFO: status: 1% (438566912/42949672960), sparse 0% (98910208), duration 53, 8/6 MB/s
201: Aug 10 00:01:55 INFO: status: 2% (859045888/42949672960), sparse 0% (113082368), duration 102, 8/8 MB/s
201: Aug 10 00:02:42 INFO: status: 3% (1288568832/42949672960), sparse 0% (127946752), duration 149, 9/8 MB/s
201: Aug 10 00:03:27 INFO: status: 4% (1718091776/42949672960), sparse 0% (130265088), duration 194, 9/9 MB/s
201: Aug 10 00:04:18 INFO: status: 5% (2155216896/42949672960), sparse 0% (130605056), duration 245, 8/8 MB/s
201: Aug 10 00:05:00 INFO: status: 6% (2582708224/42949672960), sparse 0% (134594560), duration 287, 10/10 MB/s
201: Aug 10 00:05:47 INFO: status: 7% (3010461696/42949672960), sparse 0% (143503360), duration 334, 9/8 MB/s
201: Aug 10 00:06:27 INFO: status: 8% (3447586816/42949672960), sparse 0% (146059264), duration 374, 10/10 MB/s
201: Aug 10 00:07:11 INFO: status: 9% (3865706496/42949672960), sparse 0% (208891904), duration 418, 9/8 MB/s
201: Aug 10 00:07:57 INFO: status: 10% (4302831616/42949672960), sparse 0% (211480576), duration 464, 9/9 MB/s
201: Aug 10 00:08:42 INFO: status: 11% (4732354560/42949672960), sparse 0% (217698304), duration 509, 9/9 MB/s
201: Aug 10 00:09:23 INFO: status: 12% (5154275328/42949672960), sparse 0% (218693632), duration 550, 10/10 MB/s
201: Aug 10 00:10:09 INFO: status: 13% (5591400448/42949672960), sparse 0% (221786112), duration 596, 9/9 MB/s
201: Aug 10 00:11:02 INFO: status: 14% (6017122304/42949672960), sparse 0% (225361920), duration 649, 8/7 MB/s
201: Aug 10 00:11:46 INFO: status: 15% (6453460992/42949672960), sparse 0% (231833600), duration 693, 9/9 MB/s
201: Aug 10 00:12:29 INFO: status: 16% (6872367104/42949672960), sparse 0% (245686272), duration 736, 9/9 MB/s
201: Aug 10 00:13:23 INFO: status: 17% (7309492224/42949672960), sparse 0% (247709696), duration 790, 8/8 MB/s
201: Aug 10 00:13:58 INFO: status: 18% (7735214080/42949672960), sparse 0% (248975360), duration 825, 12/12 MB/s
201: Aug 10 00:15:42 INFO: status: 19% (8160935936/42949672960), sparse 0% (253968384), duration 929, 4/4 MB/s
201: Aug 10 00:18:08 INFO: status: 20% (8592031744/42949672960), sparse 0% (267853824), duration 1075, 2/2 MB/s
201: Aug 10 00:20:35 INFO: status: 21% (9020702720/42949672960), sparse 0% (310829056), duration 1222, 2/2 MB/s
201: Aug 10 00:23:02 INFO: status: 22% (9449766912/42949672960), sparse 0% (344350720), duration 1369, 2/2 MB/s
201: Aug 10 00:26:01 INFO: status: 23% (9879617536/42949672960), sparse 0% (368967680), duration 1548, 2/2 MB/s
201: Aug 10 00:29:08 INFO: status: 24% (10310189056/42949672960), sparse 0% (384778240), duration 1735, 2/2 MB/s
201: Aug 10 00:32:25 INFO: status: 25% (10738860032/42949672960), sparse 0% (389107712), duration 1932, 2/2 MB/s
201: Aug 10 00:33:49 INFO: status: 26% (11175264256/42949672960), sparse 0% (413491200), duration 2016, 5/4 MB/s
201: Aug 10 00:34:24 INFO: status: 27% (11600461824/42949672960), sparse 1% (436903936), duration 2051, 12/11 MB/s
201: Aug 10 00:34:52 INFO: status: 28% (12032540672/42949672960), sparse 1% (660271104), duration 2079, 15/7 MB/s
201: Aug 10 00:35:19 INFO: status: 29% (12459966464/42949672960), sparse 2% (991076352), duration 2106, 15/3 MB/s
201: Aug 10 00:36:07 INFO: status: 30% (12885688320/42949672960), sparse 2% (993873920), duration 2154, 8/8 MB/s
201: Aug 10 00:36:57 INFO: status: 31% (13330415616/42949672960), sparse 2% (994508800), duration 2204, 8/8 MB/s
201: Aug 10 00:37:26 INFO: status: 32% (13752336384/42949672960), sparse 2% (995401728), duration 2233, 14/14 MB/s
201: Aug 10 00:38:08 INFO: status: 33% (14181007360/42949672960), sparse 2% (1021054976), duration 2275, 10/9 MB/s
201: Aug 10 00:38:50 INFO: status: 34% (14607581184/42949672960), sparse 2% (1029668864), duration 2317, 10/9 MB/s
201: Aug 10 00:39:36 INFO: status: 35% (15034286080/42949672960), sparse 2% (1031036928), duration 2363, 9/9 MB/s
201: Aug 10 00:40:18 INFO: status: 36% (15466430464/42949672960), sparse 2% (1036451840), duration 2405, 10/10 MB/s
201: Aug 10 00:41:00 INFO: status: 37% (15892348928/42949672960), sparse 2% (1050640384), duration 2447, 10/9 MB/s
201: Aug 10 00:41:40 INFO: status: 38% (16334782464/42949672960), sparse 2% (1052188672), duration 2487, 11/11 MB/s
201: Aug 10 00:42:18 INFO: status: 39% (16754081792/42949672960), sparse 2% (1071919104), duration 2525, 11/10 MB/s
201: Aug 10 00:43:02 INFO: status: 40% (17184718848/42949672960), sparse 2% (1078734848), duration 2569, 9/9 MB/s
201: Aug 10 00:43:45 INFO: status: 41% (17618042880/42949672960), sparse 2% (1081729024), duration 2612, 10/10 MB/s
201: Aug 10 00:44:27 INFO: status: 42% (18043764736/42949672960), sparse 2% (1096667136), duration 2654, 10/9 MB/s
201: Aug 10 00:45:12 INFO: status: 43% (18477219840/42949672960), sparse 2% (1100533760), duration 2699, 9/9 MB/s
201: Aug 10 00:45:55 INFO: status: 44% (18903138304/42949672960), sparse 2% (1108316160), duration 2742, 9/9 MB/s
201: Aug 10 00:46:35 INFO: status: 45% (19328532480/42949672960), sparse 2% (1110614016), duration 2782, 10/10 MB/s
201: Aug 10 00:47:16 INFO: status: 46% (19761856512/42949672960), sparse 2% (1113780224), duration 2823, 10/10 MB/s
201: Aug 10 00:47:53 INFO: status: 47% (20194197504/42949672960), sparse 2% (1113833472), duration 2860, 11/11 MB/s
201: Aug 10 00:48:27 INFO: status: 48% (20628504576/42949672960), sparse 2% (1113956352), duration 2894, 12/12 MB/s
201: Aug 10 00:49:01 INFO: status: 49% (21050097664/42949672960), sparse 2% (1114025984), duration 2928, 12/12 MB/s
201: Aug 10 00:49:37 INFO: status: 50% (21475950592/42949672960), sparse 2% (1124491264), duration 2964, 11/11 MB/s
201: Aug 10 00:50:22 INFO: status: 51% (21909471232/42949672960), sparse 2% (1124806656), duration 3009, 9/9 MB/s
201: Aug 10 00:51:11 INFO: status: 52% (22342991872/42949672960), sparse 2% (1127178240), duration 3058, 8/8 MB/s
201: Aug 10 00:51:52 INFO: status: 53% (22768713728/42949672960), sparse 2% (1134460928), duration 3099, 10/10 MB/s
201: Aug 10 00:52:33 INFO: status: 54% (23193255936/42949672960), sparse 2% (1135206400), duration 3140, 10/10 MB/s
201: Aug 10 00:53:14 INFO: status: 55% (23627563008/42949672960), sparse 2% (1139875840), duration 3181, 10/10 MB/s
201: Aug 10 00:53:52 INFO: status: 56% (24055185408/42949672960), sparse 2% (1174302720), duration 3219, 11/10 MB/s
201: Aug 10 00:54:30 INFO: status: 57% (24482807808/42949672960), sparse 2% (1175449600), duration 3257, 11/11 MB/s
201: Aug 10 00:55:09 INFO: status: 58% (24916131840/42949672960), sparse 2% (1208561664), duration 3296, 11/10 MB/s
201: Aug 10 00:55:53 INFO: status: 59% (25341853696/42949672960), sparse 2% (1211019264), duration 3340, 9/9 MB/s
201: Aug 10 00:56:36 INFO: status: 60% (25774784512/42949672960), sparse 2% (1211924480), duration 3383, 10/10 MB/s
201: Aug 10 00:57:16 INFO: status: 61% (26208501760/42949672960), sparse 2% (1212993536), duration 3423, 10/10 MB/s
201: Aug 10 00:58:01 INFO: status: 62% (26630422528/42949672960), sparse 2% (1221734400), duration 3468, 9/9 MB/s
201: Aug 10 00:58:43 INFO: status: 63% (27060207616/42949672960), sparse 2% (1224527872), duration 3510, 10/10 MB/s
201: Aug 10 00:59:25 INFO: status: 64% (27493269504/42949672960), sparse 2% (1226579968), duration 3552, 10/10 MB/s
201: Aug 10 01:00:14 INFO: status: 65% (27929477120/42949672960), sparse 2% (1229090816), duration 3601, 8/8 MB/s
201: Aug 10 01:01:12 INFO: status: 66% (28348514304/42949672960), sparse 2% (1230274560), duration 3659, 7/7 MB/s
201: Aug 10 01:01:52 INFO: status: 67% (28781838336/42949672960), sparse 2% (1235890176), duration 3699, 10/10 MB/s
201: Aug 10 01:02:32 INFO: status: 68% (29208346624/42949672960), sparse 2% (1247084544), duration 3739, 10/10 MB/s
201: Aug 10 01:03:11 INFO: status: 69% (29644685312/42949672960), sparse 2% (1249316864), duration 3778, 11/11 MB/s
201: Aug 10 01:03:53 INFO: status: 70% (30070407168/42949672960), sparse 2% (1251467264), duration 3820, 10/10 MB/s
201: Aug 10 01:04:34 INFO: status: 71% (30499930112/42949672960), sparse 2% (1254395904), duration 3861, 10/10 MB/s
201: Aug 10 01:05:18 INFO: status: 72% (30926307328/42949672960), sparse 2% (1260482560), duration 3905, 9/9 MB/s
201: Aug 10 01:05:58 INFO: status: 73% (31360155648/42949672960), sparse 2% (1269284864), duration 3945, 10/10 MB/s
201: Aug 10 01:06:41 INFO: status: 74% (31792300032/42949672960), sparse 2% (1281069056), duration 3988, 10/9 MB/s
201: Aug 10 01:07:19 INFO: status: 75% (32214220800/42949672960), sparse 2% (1284730880), duration 4026, 11/11 MB/s
201: Aug 10 01:08:00 INFO: status: 76% (32643743744/42949672960), sparse 3% (1293959168), duration 4067, 10/10 MB/s
201: Aug 10 01:08:44 INFO: status: 77% (33073266688/42949672960), sparse 3% (1294716928), duration 4111, 9/9 MB/s
201: Aug 10 01:09:27 INFO: status: 78% (33506590720/42949672960), sparse 3% (1302179840), duration 4154, 10/9 MB/s
201: Aug 10 01:10:14 INFO: status: 79% (33932312576/42949672960), sparse 3% (1302351872), duration 4201, 9/9 MB/s
201: Aug 10 01:11:03 INFO: status: 80% (34369437696/42949672960), sparse 3% (1302499328), duration 4250, 8/8 MB/s
201: Aug 10 01:11:48 INFO: status: 81% (34796732416/42949672960), sparse 3% (1303928832), duration 4295, 9/9 MB/s
201: Aug 10 01:12:28 INFO: status: 82% (35224682496/42949672960), sparse 3% (1312518144), duration 4335, 10/10 MB/s
201: Aug 10 01:13:05 INFO: status: 83% (35654205440/42949672960), sparse 3% (1315418112), duration 4372, 11/11 MB/s
201: Aug 10 01:13:52 INFO: status: 84% (36085432320/42949672960), sparse 3% (1332334592), duration 4419, 9/8 MB/s
201: Aug 10 01:14:21 INFO: status: 85% (36517052416/42949672960), sparse 3% (1341460480), duration 4448, 14/14 MB/s
201: Aug 10 01:15:01 INFO: status: 86% (36941463552/42949672960), sparse 3% (1351892992), duration 4488, 10/10 MB/s
201: Aug 10 01:15:44 INFO: status: 87% (37368561664/42949672960), sparse 3% (1393442816), duration 4531, 9/8 MB/s
201: Aug 10 01:16:28 INFO: status: 88% (37798019072/42949672960), sparse 3% (1396506624), duration 4575, 9/9 MB/s
201: Aug 10 01:17:14 INFO: status: 89% (38235144192/42949672960), sparse 3% (1398353920), duration 4621, 9/9 MB/s
201: Aug 10 01:17:58 INFO: status: 90% (38663225344/42949672960), sparse 3% (1405112320), duration 4665, 9/9 MB/s
201: Aug 10 01:18:41 INFO: status: 91% (39086587904/42949672960), sparse 3% (1406623744), duration 4708, 9/9 MB/s
201: Aug 10 01:19:25 INFO: status: 92% (39523188736/42949672960), sparse 3% (1411305472), duration 4752, 9/9 MB/s
201: Aug 10 01:20:10 INFO: status: 93% (39953235968/42949672960), sparse 3% (1419235328), duration 4797, 9/9 MB/s
201: Aug 10 01:20:50 INFO: status: 94% (40375156736/42949672960), sparse 3% (1423249408), duration 4837, 10/10 MB/s
201: Aug 10 01:21:36 INFO: status: 95% (40808611840/42949672960), sparse 3% (1424470016), duration 4883, 9/9 MB/s
201: Aug 10 01:22:24 INFO: status: 96% (41240494080/42949672960), sparse 3% (1431785472), duration 4931, 8/8 MB/s
201: Aug 10 01:23:07 INFO: status: 97% (41668182016/42949672960), sparse 3% (1433636864), duration 4974, 9/9 MB/s
201: Aug 10 01:23:43 INFO: status: 98% (42096721920/42949672960), sparse 3% (1545793536), duration 5010, 11/8 MB/s
201: Aug 10 01:24:20 INFO: status: 99% (42530373632/42949672960), sparse 3% (1667772416), duration 5047, 11/8 MB/s
201: Aug 10 01:25:03 INFO: status: 100% (42949672960/42949672960), sparse 3% (1699946496), duration 5090, 9/9 MB/s
201: Aug 10 01:25:03 INFO: transferred 42949 MB in 5090 seconds (8 MB/s)
201: Aug 10 01:25:04 INFO: archive file size: 16.89GB
201: Aug 10 01:25:04 INFO: delete old backup '/mnt/pve/NightlyBackup/dump/vzdump-qemu-201-2014_08_02-23_39_51.vma.gz'
201: Aug 10 01:25:06 INFO: Finished Backup of VM 201 (01:24:54)
 
Hard to diagnose when problem is not persistent.

Can you guess what's common for these vms? Are they all Windows? Windows can do filesystem optimisations which might make it more difficult to use gzip (good compression). Try lzo.

What if you try to backup some vm's on local storage? Backup them separately excluding from main backup job.
 
Hi Danielmash,

I have a mixture of linux and windows machines, windows 2008 R2. The local storage is a raid 5 with 3 drives of 72GB. So maybe local backup would not be a very good idea.

I will try to use lzo next time but what I do not understand why after a while the backup time goes up on random vms.

Any ideas are welcome

Cheers,

Raj
 
I just seen this :

https://pve.proxmox.com/wiki/Performance_Tweaks


[h=1]VZdump[/h] By default vzdump limited to 10000 Kbps (10 Mbps). That's why some users confuse with the slower data transfer on Proxmox even they claim 1 Gbps NIC installed. To increase the VZdump speed you need to edit:
# nano /etc/vzdump.conf
Find out:
#bwlimit: KBPS
bwlimit: 10000
Change to any value you like, to change bandwidth limit to 50 Mbps:
bwlimit: 50000


Now my /etc/vzdump says :

# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST
 
gzip Backup is very CPU intensive although it gives better compression. But when backing up lot of VMs, the time it will take to backup all of them using gzip outweighs the benefit. LZO is faster and less stressful on backup node. Try LZO as danielmash suggested for a week and see if it makes any difference.

I do not believe it is a network issue. Backup time usually goes up and down time to time for same VMs on same hardware. But the difference should not be in hours. I would also suggest check the performance log/status of the backup nas during backup to find any clue.
 
raj said:
So maybe local backup would not be a very good idea.
Click to expand...
Agreed. To dangerous for production. You might have other ideas how to localise the problem. eg. USB HDD connected temporary to work it out.

raj[COLOR=#3E3E3E said:
119: Aug 09 21:41:38 INFO: status: 2% (198311936/8606711808), sparse 1% (149630976), duration 11, 27/2 MB/s[/COLOR]
Click to expand...

Sometimes figures looks bigger the default bwlimit however it cached and balanced when goes to the network.

raj said:
I just seen this :

# nano /etc/vzdump.conf
#bwlimit: KBPS
bwlimit: 10000
Change to any value you like, to change bandwidth limit to 50 Mbps:
bwlimit: 50000
Click to expand...

By default vzdump limited to 10000 Kbps (10 Mbps). That's why some users confuse with the slower data transfer on Proxmox even they claim 1 Gbps NIC installed.
Click to expand...


I agreed with symmcom that more likely it's not a network issue.
However not worst to try to increase bwlimit next time. But one thing at the time! Threat compression and networking cases separately.


 
Last edited:
Ok I done 2 things, firstly did a check disk on all the vms, windows and linux which made the vms a bit faster. Then i change the bandwidth settings to bandwidth limit: 80000 KB/s. Now I am getting quite a good time for all the backups within 7 hrs and 50 mins for 100gb worth on gzip. So i wonder if the server was chocking cause of too little bandwidth allocated by default. I have not rebooted the host yet, will do so in a few days and then see if the performance stay te same.

Cheers,

Raj
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back