Slow backup speed in proxmox and i have changed vzdump.conf :)

[offerlam]

Hi all..

some may remember my problemes with backup01 mount not going away..

now i have another issue.. sob..

my backup NAS synology DS1513+ is connected with lacp to two lan switches.. so is the 3 proxmox nodes also using lacp..
The nas has two disc 7200 RPM in a Raid 1 configuration

backup NAS has a NFS share called backup01_v2 which im backing up too from proxmox..

The speeds are really slow compared to the links being 1Gbs...

what i have done to bring the speeds up..

I have set vzdump.conf with the following on all nodes:

root@proxmox01:~# cat /etc/vzdump.conf# vzdump default settings


#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
bwlimit: 900000
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#maxfiles: N
#script: FILENAME
#exclude-path: PATHLIST

I have made sure write chache is enabled on the synology NAS

Here is information gathered from the first backup attempt..bear in mind this was before i knew about vzdump.conf

VM 113

fdisk information

dingit@mail01:~$ sudo fdisk -l[sudo] password for dingit:


Disk /dev/vda: 1099.5 GB, 1099511627776 bytes
16 heads, 63 sectors/track, 2130440 cylinders, total 2147483648 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d40d9


   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048      499711      248832   83  Linux
/dev/vda2          501758  2147481599  1073489921    5  Extended
/dev/vda5          501760  2147481599  1073489920   8e  Linux LVM


Disk /dev/mapper/mail01--vg-root: 1095.0 GB, 1094956613632 bytes
255 heads, 63 sectors/track, 133120 cylinders, total 2138587136 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/mail01--vg-root doesn't contain a valid partition table


Disk /dev/mapper/mail01--vg-swap_1: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/mail01--vg-swap_1 doesn't contain a valid partition table

pveperf run from the node hosting VM113

root@proxmox00:~# pveperf /mnt/pve/Backup01_v2/CPU BOGOMIPS:      67200.32
REGEX/SECOND:      648884
HD SIZE:           3663.09 GB (10.10.98.2:/volume1/Backup01)
FSYNCS/SECOND:     536.93
DNS EXT:           46.34 ms
DNS INT:           29.81 ms (ins.local)
root@proxmox00:~# pveperf /mnt/pve/Backup01_v2/
CPU BOGOMIPS:      67200.32
REGEX/SECOND:      626250
HD SIZE:           3663.09 GB (10.10.98.2:/volume1/Backup01)
FSYNCS/SECOND:     530.65
DNS EXT:           55.97 ms
DNS INT:           50.63 ms (ins.local)

backup information from proxmox - EDIT I had to cut in this information case my post was too long.. but the backup runs steady like the second one..

[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 113 --remove 0 --mode snapshot --compress lzo --storage Backup01_v2 --node proxmox00[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 113 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: update VM 113: -lock backup[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: snapshot[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/mnt/pve/Backup01_v2/dump/vzdump-qemu-113-2014_02_04-04_35_31.vma.lzo'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: started backup task 'aaa9f842-ffd9-4da3-aed7-5c18dd2b33b8'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 0% (207749120/1099511627776), sparse 0% (133689344), duration 3, 69/24 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 1% (11015684096/1099511627776), sparse 0% (5065052160), duration 191, 57/31 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 2% (21996896256/1099511627776), sparse 1% (16042831872), duration 356, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 3% (32988790784/1099511627776), sparse 2% (27032342528), duration 530, 63/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 4% (43986976768/1099511627776), sparse 3% (38028881920), duration 717, 58/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 5% (55012884480/1099511627776), sparse 4% (49052340224), duration 891, 63/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 6% (65988788224/1099511627776), sparse 5% (60026064896), duration 1059, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 7% (76998377472/1099511627776), sparse 6% (71032815616), duration 1247, 58/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 8% (88021008384/1099511627776), sparse 7% (80991887360), duration 1431, 59/5 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 9% (98960080896/1099511627776), sparse 8% (91344314368), duration 1624, 56/3 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 10% (109984874496/1099511627776), sparse 9% (102367248384), duration 1803, 61/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 11% (120968314880/1099511627776), sparse 10% (113348571136), duration 1977, 63/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 12% (131982622720/1099511627776), sparse 11% (124361023488), duration 2151, 63/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 67% (736731201536/1099511627776), sparse 65% (725662580736), duration 11409, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 68% (747692752896/1099511627776), sparse 66% (736624095232), duration 11576, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 69% (758671998976/1099511627776), sparse 67% (747603320832), duration 11743, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 70% (769705312256/1099511627776), sparse 68% (758636613632), duration 11910, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 71% (780658999296/1099511627776), sparse 69% (769590251520), duration 12078, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 72% (791672782848/1099511627776), sparse 70% (780604014592), duration 12247, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 73% (802668871680/1099511627776), sparse 71% (791600082944), duration 12414, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 74% (813686915072/1099511627776), sparse 72% (802618064896), duration 12581, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 75% (824681365504/1099511627776), sparse 73% (813612494848), duration 12747, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 76% (835645341696/1099511627776), sparse 74% (824576450560), duration 12913, 66/0 MB/s[/FONT][/COLOR]

[COLOR=#000000][FONT=tahoma]INFO: status: 82% (901635309568/1099511627776), sparse 80% (890566004736), duration 13913, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 83% (912625106944/1099511627776), sparse 81% (901555781632), duration 14078, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 84% (923610841088/1099511627776), sparse 82% (912541495296), duration 14244, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 85% (934602080256/1099511627776), sparse 83% (922677551104), duration 14412, 65/5 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 86% (945629036544/1099511627776), sparse 84% (932618223616), duration 14581, 65/6 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 87% (956613722112/1099511627776), sparse 85% (943602888704), duration 14747, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 88% (967577108480/1099511627776), sparse 86% (954566242304), duration 14914, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 89% (978590957568/1099511627776), sparse 87% (965580070912), duration 15081, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 90% (989590061056/1099511627776), sparse 88% (976477589504), duration 15248, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 91% (1000596635648/1099511627776), sparse 89% (987301646336), duration 15415, 65/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 92% (1011550715904/1099511627776), sparse 90% (998255706112), duration 15580, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 93% (1022598053888/1099511627776), sparse 91% (1009303019520), duration 15746, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 94% (1033592373248/1099511627776), sparse 92% (1020297318400), duration 15910, 67/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 95% (1044549140480/1099511627776), sparse 93% (1031254056960), duration 16074, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 96% (1055559647232/1099511627776), sparse 94% (1042264543232), duration 16241, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 97% (1066554687488/1099511627776), sparse 95% (1053259546624), duration 16406, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 98% (1077571289088/1099511627776), sparse 96% (1064276127744), duration 16573, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 99% (1088548241408/1099511627776), sparse 97% (1075253059584), duration 16737, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 100% (1099511627776/1099511627776), sparse 98% (1086127763456), duration 16903, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: transferred 1099511 MB in 16903 seconds (65 MB/s)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: archive file size: 6.66GB[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Finished Backup of VM 113 (04:41:45)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Backup job finished successfully[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK OK[/FONT][/COLOR]

second attempt WITH vzdump.conf badwith set too 900Mbts

Proxmox backup information

[COLOR=#000000][FONT=tahoma]INFO: starting new backup job: vzdump 113 --remove 0 --mode snapshot --compress lzo --storage Backup01_v2 --node proxmox00[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Starting Backup of VM 113 (qemu)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status = running[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: update VM 113: -lock backup[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: backup mode: snapshot[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: bandwidth limit: 900000 KB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: ionice priority: 7[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: creating archive '/mnt/pve/Backup01_v2/dump/vzdump-qemu-113-2014_02_05-09_36_56.vma.lzo'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: started backup task 'c8f12be7-4763-4a17-9a2b-ea5fa162e1fe'[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 0% (186449920/1099511627776), sparse 0% (108462080), duration 3, 62/25 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 1% (11044126720/1099511627776), sparse 0% (5100593152), duration 197, 55/30 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 2% (22013214720/1099511627776), sparse 1% (16064946176), duration 363, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 3% (33039187968/1099511627776), sparse 2% (27086389248), duration 530, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 4% (44009848832/1099511627776), sparse 3% (38054739968), duration 692, 67/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 5% (54994141184/1099511627776), sparse 4% (49035771904), duration 858, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 6% (66037678080/1099511627776), sparse 5% (60077203456), duration 1025, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 7% (76978061312/1099511627776), sparse 6% (71014961152), duration 1190, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 8% (87985029120/1099511627776), sparse 7% (80863813632), duration 1361, 64/6 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 9% (98992717824/1099511627776), sparse 8% (91279728640), duration 1529, 65/3 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 10% (110011351040/1099511627776), sparse 9% (102296240128), duration 1695, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 11% (120972967936/1099511627776), sparse 10% (113255870464), duration 1863, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 12% (132001693696/1099511627776), sparse 11% (124281757696), duration 2030, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 13% (143004008448/1099511627776), sparse 12% (135282216960), duration 2198, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 14% (153963331584/1099511627776), sparse 13% (146239291392), duration 2365, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 15% (164962697216/1099511627776), sparse 14% (157236277248), duration 2533, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 16% (175942926336/1099511627776), sparse 15% (168214302720), duration 2700, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 17% (186923941888/1099511627776), sparse 16% (179192938496), duration 2868, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 18% (197919506432/1099511627776), sparse 17% (190185054208), duration 3033, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 19% (208937156608/1099511627776), sparse 18% (201200381952), duration 3202, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 20% (219926560768/1099511627776), sparse 19% (211999264768), duration 3372, 64/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 21% (230948798464/1099511627776), sparse 20% (222990086144), duration 3539, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 22% (241903861760/1099511627776), sparse 21% (233718255616), duration 3705, 65/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 23% (252897001472/1099511627776), sparse 22% (244668846080), duration 3871, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 24% (263944601600/1099511627776), sparse 23% (255713300480), duration 4038, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 25% (274907725824/1099511627776), sparse 24% (266673520640), duration 4205, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 26% (285893656576/1099511627776), sparse 25% (277657202688), duration 4371, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 27% (296913862656/1099511627776), sparse 26% (288674820096), duration 4538, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 28% (307909623808/1099511627776), sparse 27% (299668455424), duration 4705, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 29% (318916460544/1099511627776), sparse 28% (310673043456), duration 4874, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 30% (329913270272/1099511627776), sparse 29% (321667203072), duration 5039, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 31% (340853063680/1099511627776), sparse 30% (332604866560), duration 5206, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 32% (351903547392/1099511627776), sparse 31% (343653232640), duration 5372, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 33% (362879188992/1099511627776), sparse 32% (353541529600), duration 5542, 64/6 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 34% (373862432768/1099511627776), sparse 33% (364522917888), duration 5708, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 35% (384886898688/1099511627776), sparse 34% (375545458688), duration 5874, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 36% (395858083840/1099511627776), sparse 35% (386514386944), duration 6039, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 37% (406879338496/1099511627776), sparse 36% (397533523968), duration 6203, 67/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 38% (417872543744/1099511627776), sparse 37% (408501096448), duration 6370, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 39% (428848971776/1099511627776), sparse 38% (419469447168), duration 6537, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 40% (439811833856/1099511627776), sparse 39% (430430322688), duration 6702, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 41% (450855043072/1099511627776), sparse 40% (441471545344), duration 6870, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 42% (461839466496/1099511627776), sparse 41% (452443807744), duration 7038, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 43% (472855281664/1099511627776), sparse 42% (463457566720), duration 7203, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 44% (483785244672/1099511627776), sparse 43% (474385633280), duration 7366, 67/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 45% (494783954944/1099511627776), sparse 44% (484275113984), duration 7539, 63/6 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 46% (505824215040/1099511627776), sparse 45% (495313453056), duration 7708, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 47% (516773052416/1099511627776), sparse 46% (506260238336), duration 7875, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 48% (527808266240/1099511627776), sparse 47% (517293465600), duration 8041, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 49% (538816872448/1099511627776), sparse 48% (528298106880), duration 8205, 67/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 50% (549764530176/1099511627776), sparse 49% (539209498624), duration 8370, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 51% (560774119424/1099511627776), sparse 50% (550219055104), duration 8536, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 52% (571814051840/1099511627776), sparse 51% (561258967040), duration 8702, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 53% (582804504576/1099511627776), sparse 52% (572249399296), duration 8867, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 54% (593800003584/1099511627776), sparse 53% (583062786048), duration 9034, 65/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 55% (604796289024/1099511627776), sparse 54% (593903202304), duration 9201, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 56% (615770554368/1099511627776), sparse 54% (604659306496), duration 9367, 66/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 57% (626764152832/1099511627776), sparse 55% (615610322944), duration 9533, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 58% (637782130688/1099511627776), sparse 56% (626626973696), duration 9701, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 59% (648720023552/1099511627776), sparse 57% (637564846080), duration 9868, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 60% (659752026112/1099511627776), sparse 58% (648596824064), duration 10034, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 61% (670742937600/1099511627776), sparse 59% (659587715072), duration 10201, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 62% (681721659392/1099511627776), sparse 60% (670566416384), duration 10366, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 63% (692737212416/1099511627776), sparse 61% (681581948928), duration 10530, 67/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 64% (703735201792/1099511627776), sparse 62% (692579905536), duration 10696, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 65% (714691379200/1099511627776), sparse 63% (703536062464), duration 10862, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 66% (725710471168/1099511627776), sparse 64% (714555133952), duration 11029, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 67% (736723337216/1099511627776), sparse 65% (725567967232), duration 11198, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 68% (747729256448/1099511627776), sparse 66% (736573849600), duration 11364, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 69% (758669967360/1099511627776), sparse 67% (747514540032), duration 11530, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 70% (769722941440/1099511627776), sparse 68% (758567493632), duration 11695, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 71% (780695371776/1099511627776), sparse 69% (769539874816), duration 11860, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 72% (791667408896/1099511627776), sparse 70% (780511891456), duration 12025, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 73% (802653929472/1099511627776), sparse 71% (791498391552), duration 12189, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 74% (813665222656/1099511627776), sparse 72% (802509623296), duration 12354, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 75% (824645386240/1099511627776), sparse 73% (813489766400), duration 12520, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 76% (835661725696/1099511627776), sparse 74% (824506081280), duration 12688, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 77% (846629765120/1099511627776), sparse 75% (835474100224), duration 12853, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 78% (857664782336/1099511627776), sparse 76% (846509080576), duration 13019, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 79% (868658511872/1099511627776), sparse 77% (857502789632), duration 13186, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 80% (879655124992/1099511627776), sparse 78% (868499382272), duration 13351, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 81% (890614448128/1099511627776), sparse 79% (879458414592), duration 13517, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 82% (901646778368/1099511627776), sparse 80% (890490724352), duration 13683, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 83% (912605118464/1099511627776), sparse 81% (901449043968), duration 13849, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 84% (923614183424/1099511627776), sparse 82% (912458088448), duration 14017, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 85% (934601949184/1099511627776), sparse 83% (922536079360), duration 14187, 64/5 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 86% (945616322560/1099511627776), sparse 84% (932518760448), duration 14358, 64/6 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 87% (956610904064/1099511627776), sparse 85% (943513321472), duration 14525, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 88% (967606009856/1099511627776), sparse 86% (954508394496), duration 14693, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 89% (978607734784/1099511627776), sparse 87% (965510098944), duration 14860, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 90% (989596352512/1099511627776), sparse 88% (976297099264), duration 15027, 65/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 91% (1000607645696/1099511627776), sparse 89% (987125874688), duration 15194, 65/1 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 92% (1011594821632/1099511627776), sparse 90% (998113030144), duration 15359, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 93% (1022587240448/1099511627776), sparse 91% (1009105424384), duration 15527, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 94% (1033598926848/1099511627776), sparse 92% (1020117090304), duration 15696, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 95% (1044540358656/1099511627776), sparse 93% (1031058493440), duration 15860, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 96% (1055571378176/1099511627776), sparse 94% (1042089492480), duration 16026, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 97% (1066575986688/1099511627776), sparse 95% (1053094064128), duration 16193, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 98% (1077570174976/1099511627776), sparse 96% (1064088231936), duration 16360, 65/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 99% (1088518750208/1099511627776), sparse 97% (1075036786688), duration 16525, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: status: 100% (1099511627776/1099511627776), sparse 98% (1085940981760), duration 16691, 66/0 MB/s[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: transferred 1099511 MB in 16691 seconds (65 MB/s)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: archive file size: 6.71GB[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Finished Backup of VM 113 (04:38:12)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: Backup job finished successfully[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK OK[/FONT][/COLOR]

also this time i gatherd som performance data from the synology from screenshots

Network



Disk



I have read in another post that NFS had a bug in kernel version 3 dot something regarding NFS performance?? I don't know if thats what im hit by in this case?? im currently running the latest version of proxmox. We are planing to buy subscriptions for the nodes.. i don't know if that will help this issue if it is a kernel problem?

 

[offerlam]

So I added the servers local storage as a backup storage just to test the speed if the server had to backup to itself...

the VMs are stored on a second nas...

I figured if I had super speeds in this scenario i was not a proxmox problem..

here is the backup log

INFO: starting new backup job: vzdump 113 --remove 0 --mode snapshot --compress lzo --storage local --node proxmox01
INFO: Starting Backup of VM 113 (qemu)
INFO: status = running
INFO: update VM 113: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-113-2014_02_06-15_32_28.vma.lzo'
INFO: started backup task 'db28681c-02cb-479b-852b-7bdc072dc233'
INFO: status: 0% (192282624/1099511627776), sparse 0% (118034432), duration 3, 64/24 MB/s
INFO: status: 1% (11005788160/1099511627776), sparse 0% (5057376256), duration 171, 64/34 MB/s
INFO: status: 2% (22051225600/1099511627776), sparse 1% (16098267136), duration 336, 66/0 MB/s
INFO: status: 3% (33019396096/1099511627776), sparse 2% (27063263232), duration 501, 66/0 MB/s
INFO: status: 4% (44014239744/1099511627776), sparse 3% (38055858176), duration 666, 66/0 MB/s
INFO: status: 5% (55007576064/1099511627776), sparse 4% (49043632128), duration 833, 65/0 MB/s
INFO: status: 6% (66012381184/1099511627776), sparse 5% (60044644352), duration 999, 66/0 MB/s
INFO: status: 7% (77011943424/1099511627776), sparse 6% (71042154496), duration 1166, 65/0 MB/s
INFO: status: 8% (87970283520/1099511627776), sparse 7% (80832897024), duration 1331, 66/7 MB/s
INFO: status: 9% (99009953792/1099511627776), sparse 8% (91282706432), duration 1497, 66/3 MB/s
INFO: status: 10% (110014758912/1099511627776), sparse 9% (102267486208), duration 1662, 66/0 MB/s
INFO: status: 11% (120971067392/1099511627776), sparse 10% (113221939200), duration 1829, 65/0 MB/s
INFO: status: 12% (131985375232/1099511627776), sparse 11% (124233621504), duration 1994, 66/0 MB/s
INFO: status: 13% (142974582784/1099511627776), sparse 12% (135220514816), duration 2159, 66/0 MB/s
INFO: status: 14% (153951404032/1099511627776), sparse 13% (146195230720), duration 2327, 65/0 MB/s
INFO: status: 15% (164949262336/1099511627776), sparse 14% (157180461056), duration 2492, 66/0 MB/s
INFO: status: 16% (175937290240/1099511627776), sparse 15% (168165695488), duration 2657, 66/0 MB/s
INFO: status: 17% (186947403776/1099511627776), sparse 16% (179173756928), duration 2822, 66/0 MB/s
INFO: status: 18% (197934317568/1099511627776), sparse 17% (190157910016), duration 2988, 66/0 MB/s
INFO: status: 19% (208941285376/1099511627776), sparse 18% (201162928128), duration 3154, 66/0 MB/s
INFO: status: 20% (219947532288/1099511627776), sparse 19% (211975888896), duration 3319, 66/1 MB/s
INFO: status: 21% (230921142272/1099511627776), sparse 20% (222919057408), duration 3487, 65/0 MB/s
INFO: status: 22% (241908449280/1099511627776), sparse 21% (233674235904), duration 3653, 66/1 MB/s
INFO: status: 23% (252910764032/1099511627776), sparse 22% (244633739264), duration 3821, 65/0 MB/s
INFO: status: 24% (263919763456/1099511627776), sparse 23% (255639789568), duration 3987, 66/0 MB/s
INFO: status: 25% (274935971840/1099511627776), sparse 24% (266654154752), duration 4152, 66/0 MB/s
INFO: status: 26% (285883432960/1099511627776), sparse 25% (277599367168), duration 4315, 67/0 MB/s
INFO: status: 27% (296874147840/1099511627776), sparse 26% (288588017664), duration 4483, 65/0 MB/s
INFO: status: 28% (307864076288/1099511627776), sparse 27% (299576082432), duration 4648, 66/0 MB/s
INFO: status: 29% (318888345600/1099511627776), sparse 28% (310597775360), duration 4812, 67/0 MB/s
INFO: status: 30% (329860644864/1099511627776), sparse 29% (321567580160), duration 4981, 64/0 MB/s
INFO: status: 31% (340902871040/1099511627776), sparse 30% (332607152128), duration 5149, 65/0 MB/s
INFO: status: 32% (351903219712/1099511627776), sparse 31% (343605383168), duration 5316, 65/0 MB/s
INFO: status: 33% (362840260608/1099511627776), sparse 32% (353326497792), duration 5480, 66/7 MB/s
INFO: status: 34% (373836218368/1099511627776), sparse 33% (364317388800), duration 5646, 66/0 MB/s
INFO: status: 35% (384848297984/1099511627776), sparse 34% (375327608832), duration 5813, 65/0 MB/s
INFO: status: 36% (395859460096/1099511627776), sparse 35% (386336645120), duration 5980, 65/0 MB/s
INFO: status: 37% (406874619904/1099511627776), sparse 36% (397349752832), duration 6146, 66/0 MB/s
INFO: status: 38% (417824309248/1099511627776), sparse 37% (408274006016), duration 6311, 66/0 MB/s
INFO: status: 39% (428827738112/1099511627776), sparse 38% (419269292032), duration 6476, 66/0 MB/s
INFO: status: 40% (439872913408/1099511627776), sparse 39% (430312677376), duration 6641, 66/0 MB/s
INFO: status: 41% (450846916608/1099511627776), sparse 40% (441284694016), duration 6803, 67/0 MB/s
INFO: status: 42% (461809909760/1099511627776), sparse 41% (452237479936), duration 6967, 66/0 MB/s
INFO: status: 43% (472836997120/1099511627776), sparse 42% (463262711808), duration 7133, 66/0 MB/s
INFO: status: 44% (483839049728/1099511627776), sparse 43% (474262970368), duration 7300, 65/0 MB/s
INFO: status: 45% (494840709120/1099511627776), sparse 44% (484155924480), duration 7466, 66/6 MB/s
INFO: status: 46% (505835487232/1099511627776), sparse 45% (495148519424), duration 7634, 65/0 MB/s
INFO: status: 47% (516814012416/1099511627776), sparse 46% (506124599296), duration 7799, 66/0 MB/s
INFO: status: 48% (527803285504/1099511627776), sparse 47% (517112016896), duration 7966, 65/0 MB/s
INFO: status: 49% (538789675008/1099511627776), sparse 48% (528094310400), duration 8131, 66/0 MB/s
INFO: status: 50% (549767544832/1099511627776), sparse 49% (539038339072), duration 8298, 65/0 MB/s
INFO: status: 51% (560778706944/1099511627776), sparse 50% (550049402880), duration 8465, 65/0 MB/s
INFO: status: 52% (571814445056/1099511627776), sparse 51% (561085116416), duration 8630, 66/0 MB/s
INFO: status: 53% (582742835200/1099511627776), sparse 52% (572013486080), duration 8794, 66/0 MB/s
INFO: status: 54% (593782898688/1099511627776), sparse 53% (582871437312), duration 8959, 66/1 MB/s
INFO: status: 55% (604758736896/1099511627776), sparse 53% (593691500544), duration 9123, 66/0 MB/s
INFO: status: 56% (615772127232/1099511627776), sparse 54% (604486729728), duration 9288, 66/1 MB/s
INFO: status: 57% (626734727168/1099511627776), sparse 55% (615406747648), duration 9455, 65/0 MB/s
INFO: status: 58% (637755654144/1099511627776), sparse 56% (626426347520), duration 9621, 66/0 MB/s
INFO: status: 59% (648778285056/1099511627776), sparse 57% (637448892416), duration 9788, 66/0 MB/s
INFO: status: 60% (659744227328/1099511627776), sparse 58% (648414744576), duration 9951, 67/0 MB/s
INFO: status: 61% (670760632320/1099511627776), sparse 59% (659431129088), duration 10114, 67/0 MB/s
INFO: status: 62% (681758752768/1099511627776), sparse 60% (670429229056), duration 10280, 66/0 MB/s
INFO: status: 63% (692748419072/1099511627776), sparse 61% (681418874880), duration 10444, 67/0 MB/s
INFO: status: 64% (703711412224/1099511627776), sparse 62% (692381835264), duration 10606, 67/0 MB/s
INFO: status: 65% (714689478656/1099511627776), sparse 63% (703359750144), duration 10770, 66/0 MB/s
INFO: status: 66% (725686091776/1099511627776), sparse 64% (714356277248), duration 10935, 66/0 MB/s
INFO: status: 67% (736684474368/1099511627776), sparse 65% (725354627072), duration 11103, 65/0 MB/s
INFO: status: 68% (747677941760/1099511627776), sparse 66% (736348057600), duration 11268, 66/0 MB/s
INFO: status: 69% (758702800896/1099511627776), sparse 67% (747372830720), duration 11436, 65/0 MB/s
INFO: status: 70% (769691746304/1099511627776), sparse 68% (758361755648), duration 11601, 66/0 MB/s
INFO: status: 71% (780675186688/1099511627776), sparse 69% (769345146880), duration 11767, 66/0 MB/s
INFO: status: 72% (791698735104/1099511627776), sparse 70% (780368674816), duration 11931, 67/0 MB/s
INFO: status: 73% (802679160832/1099511627776), sparse 71% (791349080064), duration 12095, 66/0 MB/s
INFO: status: 74% (813649887232/1099511627776), sparse 72% (802319745024), duration 12260, 66/0 MB/s
INFO: status: 75% (824650235904/1099511627776), sparse 73% (813320073216), duration 12424, 67/0 MB/s
INFO: status: 76% (835645669376/1099511627776), sparse 74% (824315486208), duration 12589, 66/0 MB/s
INFO: status: 77% (846625243136/1099511627776), sparse 75% (835294969856), duration 12756, 65/0 MB/s
INFO: status: 78% (857639550976/1099511627776), sparse 76% (846309240832), duration 12921, 66/0 MB/s
INFO: status: 79% (868648157184/1099511627776), sparse 77% (857317826560), duration 13086, 66/0 MB/s
INFO: status: 80% (879663316992/1099511627776), sparse 78% (868332965888), duration 13250, 67/0 MB/s
INFO: status: 81% (890640793600/1099511627776), sparse 79% (879310151680), duration 13415, 66/0 MB/s
INFO: status: 82% (901649727488/1099511627776), sparse 80% (890319065088), duration 13579, 67/0 MB/s
INFO: status: 83% (912609443840/1099511627776), sparse 81% (901278760960), duration 13745, 66/0 MB/s
INFO: status: 84% (923597209600/1099511627776), sparse 82% (912266506240), duration 13910, 66/0 MB/s
INFO: status: 85% (934584975360/1099511627776), sparse 83% (922399088640), duration 14078, 65/5 MB/s
INFO: status: 86% (945612521472/1099511627776), sparse 84% (932340350976), duration 14245, 66/6 MB/s
INFO: status: 87% (956634365952/1099511627776), sparse 85% (943362174976), duration 14412, 65/0 MB/s
INFO: status: 88% (967640285184/1099511627776), sparse 86% (954368061440), duration 14577, 66/0 MB/s
INFO: status: 89% (978613764096/1099511627776), sparse 87% (965341454336), duration 14743, 66/0 MB/s
INFO: status: 90% (989610967040/1099511627776), sparse 88% (976146649088), duration 14910, 65/1 MB/s
INFO: status: 91% (1000582610944/1099511627776), sparse 89% (986935775232), duration 15077, 65/1 MB/s
INFO: status: 92% (1011600130048/1099511627776), sparse 90% (997953273856), duration 15243, 66/0 MB/s
INFO: status: 93% (1022598184960/1099511627776), sparse 91% (1008951304192), duration 15408, 66/0 MB/s
INFO: status: 94% (1033594994688/1099511627776), sparse 92% (1019948093440), duration 15572, 67/0 MB/s
INFO: status: 95% (1044561133568/1099511627776), sparse 93% (1030914203648), duration 15735, 67/0 MB/s
INFO: status: 96% (1055588286464/1099511627776), sparse 94% (1041941336064), duration 15901, 66/0 MB/s
INFO: status: 97% (1066586800128/1099511627776), sparse 95% (1052939812864), duration 16068, 65/0 MB/s
INFO: status: 98% (1077582036992/1099511627776), sparse 96% (1063935029248), duration 16234, 66/0 MB/s
INFO: status: 99% (1088562003968/1099511627776), sparse 97% (1074914975744), duration 16400, 66/0 MB/s
INFO: status: 100% (1099511627776/1099511627776), sparse 98% (1085775917056), duration 16567, 65/0 MB/s
INFO: transferred 1099511 MB in 16567 seconds (66 MB/s)
INFO: archive file size: 6.79GB
INFO: Finished Backup of VM 113 (04:36:09)
INFO: Backup job finished successfully
TASK OK

the speeds are almost identical so it must be a proxmox problem..

if 66 is the avarage transfere speed and its MB/s thats about ½ a Gigabit.. just got no clue where to look??

THANKSCasper

 

[offerlam]

As I said before i have another NAS storing VMs and such.. also nfs.. so i made a backup nfs share on that and did the backup to see if it gave me the same result

here is the log:

INFO: starting new backup job: vzdump 113 --remove 0 --mode snapshot --compress lzo --storage Storage01_Templates --node proxmox01
INFO: Starting Backup of VM 113 (qemu)
INFO: status = running
INFO: update VM 113: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Storage01_Templates/dump/vzdump-qemu-113-2014_02_06-20_17_53.vma.lzo'
INFO: started backup task 'e236bd09-bc0a-4309-945b-0fd4e92eb974'
INFO: status: 0% (207290368/1099511627776), sparse 0% (133046272), duration 3, 69/24 MB/s
INFO: status: 1% (11031216128/1099511627776), sparse 0% (5078454272), duration 200, 54/29 MB/s
INFO: status: 2% (21993750528/1099511627776), sparse 1% (16038268928), duration 365, 66/0 MB/s
INFO: status: 3% (32999014400/1099511627776), sparse 2% (27041144832), duration 531, 66/0 MB/s
INFO: status: 4% (43984748544/1099511627776), sparse 3% (38025089024), duration 695, 66/0 MB/s
INFO: status: 5% (55016161280/1099511627776), sparse 4% (49053396992), duration 859, 67/0 MB/s
INFO: status: 6% (66027061248/1099511627776), sparse 5% (60062441472), duration 1024, 66/0 MB/s
INFO: status: 7% (76982255616/1099511627776), sparse 6% (71015845888), duration 1189, 66/0 MB/s
INFO: status: 8% (87991844864/1099511627776), sparse 7% (80859717632), duration 1358, 65/6 MB/s
INFO: status: 9% (98974171136/1099511627776), sparse 8% (91247505408), duration 1524, 66/3 MB/s
INFO: status: 10% (109951320064/1099511627776), sparse 9% (102222860288), duration 1689, 66/0 MB/s
INFO: status: 11% (120953110528/1099511627776), sparse 10% (113222729728), duration 1856, 65/0 MB/s
INFO: status: 12% (131998744576/1099511627776), sparse 11% (124266377216), duration 2020, 67/0 MB/s
INFO: status: 13% (143000141824/1099511627776), sparse 12% (135265800192), duration 2184, 67/0 MB/s
INFO: status: 14% (153944981504/1099511627776), sparse 13% (146208522240), duration 2350, 65/0 MB/s
INFO: status: 15% (164967350272/1099511627776), sparse 14% (157229363200), duration 2516, 66/0 MB/s
INFO: status: 16% (175941615616/1099511627776), sparse 15% (168201555968), duration 2680, 66/0 MB/s
INFO: status: 17% (186964049920/1099511627776), sparse 16% (179222134784), duration 2845, 66/0 MB/s
INFO: status: 18% (197959876608/1099511627776), sparse 17% (190215692288), duration 3010, 66/0 MB/s
INFO: status: 19% (208978313216/1099511627776), sparse 18% (201232003072), duration 3176, 66/0 MB/s
INFO: status: 20% (219966275584/1099511627776), sparse 19% (212023668736), duration 3343, 65/1 MB/s
INFO: status: 21% (230906724352/1099511627776), sparse 20% (222933487616), duration 3506, 67/0 MB/s
INFO: status: 22% (241934008320/1099511627776), sparse 21% (233725693952), duration 3672, 66/1 MB/s
INFO: status: 23% (252915286016/1099511627776), sparse 22% (244664487936), duration 3836, 66/0 MB/s
INFO: status: 24% (263947550720/1099511627776), sparse 23% (255693934592), duration 4000, 67/0 MB/s
INFO: status: 25% (274917949440/1099511627776), sparse 24% (266662543360), duration 4163, 67/0 MB/s
INFO: status: 26% (285881860096/1099511627776), sparse 25% (277624664064), duration 4326, 67/0 MB/s
INFO: status: 27% (296894464000/1099511627776), sparse 26% (288634875904), duration 4490, 67/0 MB/s
INFO: status: 28% (307931578368/1099511627776), sparse 27% (299670126592), duration 4655, 66/0 MB/s
INFO: status: 29% (318898176000/1099511627776), sparse 28% (310634409984), duration 4817, 67/0 MB/s
INFO: status: 30% (329880502272/1099511627776), sparse 29% (321614675968), duration 4976, 69/0 MB/s
INFO: status: 31% (340893040640/1099511627776), sparse 30% (332625477632), duration 5140, 67/0 MB/s
INFO: status: 32% (351882379264/1099511627776), sparse 31% (343612895232), duration 5303, 67/0 MB/s
INFO: status: 33% (362876633088/1099511627776), sparse 32% (353387855872), duration 5473, 64/7 MB/s
INFO: status: 34% (373893103616/1099511627776), sparse 33% (364402667520), duration 5638, 66/0 MB/s
INFO: status: 35% (384837025792/1099511627776), sparse 34% (375344795648), duration 5802, 66/0 MB/s
INFO: status: 36% (395858214912/1099511627776), sparse 35% (386364252160), duration 5965, 67/0 MB/s
INFO: status: 37% (406881828864/1099511627776), sparse 36% (397386141696), duration 6131, 66/0 MB/s
INFO: status: 38% (417818869760/1099511627776), sparse 37% (408297484288), duration 6294, 67/0 MB/s
INFO: status: 39% (428840321024/1099511627776), sparse 38% (419311251456), duration 6459, 66/0 MB/s
INFO: status: 40% (439844798464/1099511627776), sparse 39% (430314135552), duration 6621, 67/0 MB/s
INFO: status: 41% (450843049984/1099511627776), sparse 40% (441310531584), duration 6786, 66/0 MB/s
INFO: status: 42% (461808795648/1099511627776), sparse 41% (452264247296), duration 6951, 66/0 MB/s
INFO: status: 43% (472793088000/1099511627776), sparse 42% (463246548992), duration 7117, 66/0 MB/s
INFO: status: 44% (483797762048/1099511627776), sparse 43% (474249695232), duration 7282, 66/0 MB/s
INFO: status: 45% (494801846272/1099511627776), sparse 44% (484145205248), duration 7450, 65/6 MB/s
INFO: status: 46% (505779650560/1099511627776), sparse 45% (495121219584), duration 7615, 66/0 MB/s
INFO: status: 47% (516772593664/1099511627776), sparse 46% (506112110592), duration 7778, 67/0 MB/s
INFO: status: 48% (527781527552/1099511627776), sparse 47% (517119320064), duration 7939, 68/0 MB/s
INFO: status: 49% (538814709760/1099511627776), sparse 48% (528148668416), duration 8103, 67/0 MB/s
INFO: status: 50% (549771804672/1099511627776), sparse 49% (539057831936), duration 8266, 67/0 MB/s
INFO: status: 51% (560760815616/1099511627776), sparse 50% (550046806016), duration 8431, 66/0 MB/s
INFO: status: 52% (571772567552/1099511627776), sparse 51% (561058537472), duration 8594, 67/0 MB/s
INFO: status: 53% (582741590016/1099511627776), sparse 52% (572027539456), duration 8760, 66/0 MB/s
INFO: status: 54% (593770774528/1099511627776), sparse 53% (582874611712), duration 8923, 67/1 MB/s
INFO: status: 55% (604768174080/1099511627776), sparse 53% (593715908608), duration 9087, 67/0 MB/s
INFO: status: 56% (615776059392/1099511627776), sparse 54% (604505632768), duration 9254, 65/1 MB/s
INFO: status: 57% (626767626240/1099511627776), sparse 55% (615454617600), duration 9418, 67/0 MB/s
INFO: status: 58% (637752049664/1099511627776), sparse 56% (626437713920), duration 9584, 66/0 MB/s
INFO: status: 59% (648767209472/1099511627776), sparse 57% (637452787712), duration 9748, 67/0 MB/s
INFO: status: 60% (659709624320/1099511627776), sparse 58% (648395177984), duration 9912, 66/0 MB/s
INFO: status: 61% (670757093376/1099511627776), sparse 59% (659442626560), duration 10078, 66/0 MB/s
INFO: status: 62% (681711566848/1099511627776), sparse 60% (670397079552), duration 10241, 67/0 MB/s
INFO: status: 63% (692700119040/1099511627776), sparse 61% (681385611264), duration 10406, 66/0 MB/s
INFO: status: 64% (703735922688/1099511627776), sparse 62% (692421382144), duration 10571, 66/0 MB/s
INFO: status: 65% (714698129408/1099511627776), sparse 63% (703383568384), duration 10734, 67/0 MB/s
INFO: status: 66% (725727379456/1099511627776), sparse 64% (714412797952), duration 10899, 66/0 MB/s
INFO: status: 67% (736681066496/1099511627776), sparse 65% (725366452224), duration 11062, 67/0 MB/s
INFO: status: 68% (747682201600/1099511627776), sparse 66% (736367550464), duration 11225, 67/0 MB/s
INFO: status: 69% (758676127744/1099511627776), sparse 67% (747361456128), duration 11391, 66/0 MB/s
INFO: status: 70% (769719336960/1099511627776), sparse 68% (758404579328), duration 11555, 67/0 MB/s
INFO: status: 71% (780664832000/1099511627776), sparse 69% (769350025216), duration 11718, 67/0 MB/s
INFO: status: 72% (791669047296/1099511627776), sparse 70% (780354220032), duration 11883, 66/0 MB/s
INFO: status: 73% (802670706688/1099511627776), sparse 71% (791355858944), duration 12048, 66/0 MB/s
INFO: status: 74% (813684752384/1099511627776), sparse 72% (802369777664), duration 12215, 65/0 MB/s
INFO: status: 75% (824675139584/1099511627776), sparse 73% (813360144384), duration 12376, 68/0 MB/s
INFO: status: 76% (835657531392/1099511627776), sparse 74% (824342511616), duration 12542, 66/0 MB/s
INFO: status: 77% (846667186176/1099511627776), sparse 75% (835352145920), duration 12710, 65/0 MB/s
INFO: status: 78% (857636601856/1099511627776), sparse 76% (846321459200), duration 12875, 66/0 MB/s
INFO: status: 79% (868676665344/1099511627776), sparse 77% (857361502208), duration 13038, 67/0 MB/s
INFO: status: 80% (879646605312/1099511627776), sparse 78% (868331421696), duration 13199, 68/0 MB/s
INFO: status: 81% (890640203776/1099511627776), sparse 79% (879324729344), duration 13364, 66/0 MB/s
INFO: status: 82% (901602803712/1099511627776), sparse 80% (890287308800), duration 13528, 66/0 MB/s
INFO: status: 83% (912645488640/1099511627776), sparse 81% (901329973248), duration 13694, 66/0 MB/s
INFO: status: 84% (923610513408/1099511627776), sparse 82% (912294977536), duration 13860, 66/0 MB/s
INFO: status: 85% (934631243776/1099511627776), sparse 83% (922397544448), duration 14030, 64/5 MB/s
INFO: status: 86% (945580539904/1099511627776), sparse 84% (932323471360), duration 14198, 65/6 MB/s
INFO: status: 87% (956615360512/1099511627776), sparse 85% (943358271488), duration 14362, 67/0 MB/s
INFO: status: 88% (967617413120/1099511627776), sparse 86% (954360291328), duration 14523, 68/0 MB/s
INFO: status: 89% (978569527296/1099511627776), sparse 87% (965312385024), duration 14686, 67/0 MB/s
INFO: status: 90% (989588946944/1099511627776), sparse 88% (976052240384), duration 14853, 65/1 MB/s
INFO: status: 91% (1000583004160/1099511627776), sparse 89% (986863779840), duration 15019, 66/1 MB/s
INFO: status: 92% (1011584335872/1099511627776), sparse 90% (997865091072), duration 15181, 67/0 MB/s
INFO: status: 93% (1022577082368/1099511627776), sparse 91% (1008857812992), duration 15345, 67/0 MB/s
INFO: status: 94% (1033556787200/1099511627776), sparse 92% (1019837497344), duration 15506, 68/0 MB/s
INFO: status: 95% (1044574306304/1099511627776), sparse 93% (1030854987776), duration 15670, 67/0 MB/s
INFO: status: 96% (1055544180736/1099511627776), sparse 94% (1041824841728), duration 15833, 67/0 MB/s
INFO: status: 97% (1066576445440/1099511627776), sparse 95% (1052857069568), duration 15997, 67/0 MB/s
INFO: status: 98% (1077549924352/1099511627776), sparse 96% (1063830528000), duration 16162, 66/0 MB/s
INFO: status: 99% (1088527466496/1099511627776), sparse 97% (1074808049664), duration 16330, 65/0 MB/s
INFO: status: 100% (1099511627776/1099511627776), sparse 98% (1085703528448), duration 16493, 67/0 MB/s
INFO: transferred 1099511 MB in 16493 seconds (66 MB/s)
INFO: archive file size: 6.84GB
INFO: Finished Backup of VM 113 (04:35:02)
INFO: Backup job finished successfully
TASK OK

 

[offerlam]

Im starting to think it is my proxmox that has a problem because no matter if i do backup on

local store
nas01
nas02

I get 66MB/s or around that ... is that not wierd? its excatly half the speed of my Gbs network 500Mbits.. so I wondered if it was a half duplex thing but it it doesn't look like it

duplex on proxmox00

root@proxmox00:~# dmesg | grep -i duplex
igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth2, 1000 Mbps full duplex.
igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth3, 1000 Mbps full duplex.

duplex on proxmox01

root@proxmox01:~# dmesg | grep -i duplex
igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth2, 1000 Mbps full duplex.
igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth3, 1000 Mbps full duplex.
igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth3, 1000 Mbps full duplex.
igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth3, 1000 Mbps full duplex.

duplex on proxmox02

root@proxmox02:~# dmesg | grep -i duplex
igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth2, 1000 Mbps full duplex.
igb: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth3, 1000 Mbps full duplex.
igb: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
bond0: link status definitely up for interface eth2, 1000 Mbps full duplex.

I can't attach pictures.. the manage attachments button doesn't work.. but i have checked and both synology boxes says they are running 1000Mbits..

so if the proxmox servers and the nas are full duplex than the switch must be too.. right?...

 

[thheo]

If you are unsure about your network performance use iperf to test it..

 

[offerlam]

If you are unsure about your network performance use iperf to test it..

Hi thheo and thank for answering..

since the backup node is a synology DS1513+ I don't know how to install iperf on it.. if its even possible.. but i did do it on two proxmox nods just to see if perhaps the switch could be the issue..

here is the result

root@proxmox00:~# iperf -c 10.10.99.21 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 10.10.99.21, TCP port 5001
TCP window size:  281 KByte (default)
------------------------------------------------------------
[  5] local 10.10.99.20 port 48524 connected with 10.10.99.21 port 5001
[  4] local 10.10.99.20 port 5001 connected with 10.10.99.21 port 35645
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec  1.05 GBytes   902 Mbits/sec
[  4]  0.0-10.0 sec  1.03 GBytes   884 Mbits/sec
root@proxmox00:~#

clearly i get almost 1Gbs. According to google these speeds is the same as between 110,5 MB/s and 112,75 MB/s which i don't get with my NAS where its between 60-67 MB/s

But i did find that i had, by mistake, set tag on for jumbo frames on one switch on not the other. I haven't configured jumbo frames yet so i ticked it off... and now my backup speed to the nas dropped to an about steady 10MB/s. Now i have seen a post about this somewhere where the user had to edit vzdump.conf and change the bwlimit settings. But i belive that solution is not ment for proxmox 3.1 since everything in my vzdump.file is commented out. I think it was because in the old version of proxmox there were a bwlimit of 10Mb/s default set in the vzdump.conf..

I tested it and it did nothing so i have tried.. and i did it by setting bwlimit to 131072 which should be 1Gbit and then ran the backup. I did not restart the service or node but in the backup log i noticed that i had seen my bwlimit setting.. but it did nothing for the speed so i changed my vzdum.conf back to nothing..

ANY suggestions would be AWESOME!!! cause im really getting desperate here...

I can note that my VMs that i test this with has a 1TB virtual LVM disk to backup op.. that being said the disc is by far full which also explains why the backup when done only takes up 6.6 Gigs. But i did read somewhere that having such a big disk could cause problems or give you a wrong speed output in the backup log?? I do see in the log that the backup things it has to backup 1TB of data.. you can also see that in the my log posts further up...

THANKS

Casper

 

[offerlam]

ok now i'm sure i have a proxmox problem...

so I found this site

http://nfs.sourceforge.net/nfs-howto/ar01s05.html

where they suggest you can test your NFS speeds with this command

time dd if=/dev/zero of=/mnt/home/testfile bs=16k count=16384

to create a test file and than this to test the speed

time dd if=/mnt/home/testfile of=/dev/null bs=16k

or thats how i understand it..

they also state that

 In general, you should create a file that's at least twice as large as the system RAM on the server

and since my ds1513+ has 2GB of ram i calculated i would need to set the command to use 262144 blocks..

so i did that with the first command

root@proxmox00:~# time dd if=/dev/zero of=/mnt/pve/Backup01/testfile bs=16k count=262144
262144+0 records in
262144+0 records out
4294967296 bytes (4.3 GB) copied, 43.032 s, 99.8 MB/s

real    0m44.106s
user    0m0.091s
sys     0m6.952s
root@proxmox00:~#

and now to test the second command. They say to run it a couple of times to make sure you can get a average

root@proxmox00:~# time dd if=/mnt/pve/Backup01/testfile of=/dev/null bs=16k     262144+0 records in
262144+0 records out
4294967296 bytes (4.3 GB) copied, 40.8836 s, 105 MB/s

real    0m40.886s
user    0m0.096s
sys     0m4.132s
root@proxmox00:~# time dd if=/mnt/pve/Backup01/testfile of=/dev/null bs=16k
262144+0 records in
262144+0 records out
4294967296 bytes (4.3 GB) copied, 2.19469 s, 2.0 GB/s

real    0m2.197s
user    0m0.068s
sys     0m2.119s
root@proxmox00:~# time dd if=/mnt/pve/Backup01/testfile of=/dev/null bs=16k
262144+0 records in
262144+0 records out
4294967296 bytes (4.3 GB) copied, 1.80916 s, 2.4 GB/s

real    0m1.812s
user    0m0.065s
sys     0m1.746s
root@proxmox00:~# time dd if=/mnt/pve/Backup01/testfile of=/dev/null bs=16k
262144+0 records in
262144+0 records out
4294967296 bytes (4.3 GB) copied, 1.95482 s, 2.2 GB/s

real    0m1.957s
user    0m0.072s
sys     0m1.878s
root@proxmox00:~# time dd if=/mnt/pve/Backup01/testfile of=/dev/null bs=16k
262144+0 records in
262144+0 records out
4294967296 bytes (4.3 GB) copied, 1.83798 s, 2.3 GB/s

real    0m1.840s
user    0m0.076s
sys     0m1.761s
root@proxmox00:~#

so in the first go i get 105 MB/s and in the other I get 2.4 2.2 and 2.3 GB/s

now this stinks to high heaven that somehow after my first run proxmox chache the data which explains the insane transfere speeds after the first run... but i would say this shows that there IS a 105 MB/s speed between my proxmox node and my NAS.. and again this suggest i have a proxmox problem somewhere i just don't know where to look... HELP!!!! SOS!!! MAYDAY MAYDAY!!!

Thanks

Casper

 

[offerlam]

ok so i found a misconfiguration on one of my switches where jumbo frames had been ticked on... i removed that and booted both switches..

after that my backups now went from 66 MB/s avarage to 10 MB/s flat!!!

INFO: starting new backup job: vzdump 113 --remove 0 --mode snapshot --compress lzo --storage Backup01 --node proxmox00
INFO: Starting Backup of VM 113 (qemu)
INFO: status = running
INFO: update VM 113: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backup01/dump/vzdump-qemu-113-2014_02_07-17_31_33.vma.lzo'
INFO: started backup task '0584becb-27d9-4cb2-b323-cf0df889e474'
INFO: status: 0% (32243712/1099511627776), sparse 0% (3547136), duration 3, 10/9 MB/s
INFO: status: 1% (11001200640/1099511627776), sparse 0% (4808302592), duration 1057, 10/5 MB/s
INFO: status: 2% (21991915520/1099511627776), sparse 1% (15785193472), duration 2069, 10/0 MB/s
INFO: status: 3% (32989118464/1099511627776), sparse 2% (26770604032), duration 3080, 10/0 MB/s
INFO: status: 4% (43984093184/1099511627776), sparse 3% (37753298944), duration 4094, 10/0 MB/s
INFO: status: 5% (54978412544/1099511627776), sparse 4% (48734605312), duration 5106, 10/0 MB/s
INFO: status: 6% (65979154432/1099511627776), sparse 5% (59721695232), duration 6119, 10/0 MB/s
INFO: status: 7% (76971048960/1099511627776), sparse 6% (70700920832), duration 7132, 10/0 MB/s
INFO: status: 8% (87965237248/1099511627776), sparse 7% (80677302272), duration 8147, 10/1 MB/s
INFO: status: 9% (98958639104/1099511627776), sparse 8% (91067752448), duration 9161, 10/0 MB/s
INFO: status: 10% (109954138112/1099511627776), sparse 9% (102050582528), duration 10174, 10/0 MB/s
INFO: status: 11% (120948588544/1099511627776), sparse 10% (113033539584), duration 11186, 10/0 MB/s
INFO: status: 12% (131952279552/1099511627776), sparse 11% (124026331136), duration 12198, 10/0 MB/s
INFO: status: 13% (142946140160/1099511627776), sparse 12% (135015649280), duration 13210, 10/0 MB/s
INFO: status: 14% (153934430208/1099511627776), sparse 13% (146003591168), duration 14223, 10/0 MB/s
INFO: status: 15% (164936744960/1099511627776), sparse 14% (157005295616), duration 15236, 10/0 MB/s
INFO: status: 16% (175924772864/1099511627776), sparse 15% (167973478400), duration 16256, 10/0 MB/s
INFO: status: 17% (186917191680/1099511627776), sparse 16% (178964811776), duration 17267, 10/0 MB/s
INFO: status: 18% (197922258944/1099511627776), sparse 17% (189969399808), duration 18281, 10/0 MB/s
INFO: status: 19% (208917823488/1099511627776), sparse 18% (200964390912), duration 19293, 10/0 MB/s
INFO: status: 20% (219912798208/1099511627776), sparse 19% (211764981760), duration 20306, 10/0 MB/s
INFO: status: 21% (230900039680/1099511627776), sparse 20% (222718709760), duration 21320, 10/0 MB/s
INFO: status: 22% (241902485504/1099511627776), sparse 21% (233487130624), duration 22333, 10/0 MB/s
INFO: status: 23% (252893528064/1099511627776), sparse 22% (244437262336), duration 23347, 10/0 MB/s
INFO: status: 24% (263890141184/1099511627776), sparse 23% (255431974912), duration 24360, 10/0 MB/s
INFO: status: 25% (274888392704/1099511627776), sparse 24% (266429812736), duration 25373, 10/0 MB/s
INFO: status: 26% (285882122240/1099511627776), sparse 25% (277422931968), duration 26386, 10/0 MB/s
INFO: status: 27% (296874737664/1099511627776), sparse 26% (288414728192), duration 27398, 10/0 MB/s
INFO: status: 28% (307865059328/1099511627776), sparse 27% (299404562432), duration 28411, 10/0 MB/s
INFO: status: 29% (318864097280/1099511627776), sparse 28% (310402662400), duration 29425, 10/0 MB/s
INFO: status: 30% (329857892352/1099511627776), sparse 29% (321395314688), duration 30438, 10/0 MB/s
INFO: status: 31% (340851032064/1099511627776), sparse 30% (332387438592), duration 31450, 10/0 MB/s
INFO: status: 32% (351849873408/1099511627776), sparse 31% (343385341952), duration 32463, 10/0 MB/s
INFO: status: 33% (362843340800/1099511627776), sparse 32% (353269284864), duration 33478, 10/1 MB/s
INFO: status: 34% (373836218368/1099511627776), sparse 33% (364262141952), duration 34491, 10/0 MB/s
INFO: status: 35% (384839122944/1099511627776), sparse 34% (375265021952), duration 35506, 10/0 MB/s
INFO: status: 36% (395833835520/1099511627776), sparse 35% (386259705856), duration 36518, 10/0 MB/s
INFO: status: 37% (406821339136/1099511627776), sparse 36% (397247188992), duration 37534, 10/0 MB/s
INFO: status: 38% (417820442624/1099511627776), sparse 37% (408222625792), duration 38547, 10/0 MB/s
INFO: status: 39% (428816072704/1099511627776), sparse 38% (419212275712), duration 39561, 10/0 MB/s
INFO: status: 40% (439806394368/1099511627776), sparse 39% (430202576896), duration 40573, 10/0 MB/s
INFO: status: 41% (450804383744/1099511627776), sparse 40% (441200492544), duration 41585, 10/0 MB/s
INFO: status: 42% (461800472576/1099511627776), sparse 41% (452186374144), duration 42611, 10/0 MB/s
INFO: status: 43% (472799641600/1099511627776), sparse 42% (463185522688), duration 43625, 10/0 MB/s
INFO: status: 44% (483785310208/1099511627776), sparse 43% (474171166720), duration 44638, 10/0 MB/s
INFO: status: 45% (494781464576/1099511627776), sparse 44% (484060450816), duration 45696, 10/1 MB/s
INFO: status: 46% (505782468608/1099511627776), sparse 45% (495009722368), duration 46760, 10/0 MB/s
INFO: status: 47% (516778950656/1099511627776), sparse 46% (506006183936), duration 47822, 10/0 MB/s
INFO: status: 48% (527766585344/1099511627776), sparse 47% (516950810624), duration 48856, 10/0 MB/s
INFO: status: 49% (538766016512/1099511627776), sparse 48% (527948111872), duration 49870, 10/0 MB/s
INFO: status: 50% (549756076032/1099511627776), sparse 49% (538938081280), duration 50881, 10/0 MB/s
INFO: status: 51% (560752623616/1099511627776), sparse 50% (549934596096), duration 51894, 10/0 MB/s
INFO: status: 52% (571751399424/1099511627776), sparse 51% (560933347328), duration 52908, 10/0 MB/s
INFO: status: 53% (582745522176/1099511627776), sparse 52% (571927449600), duration 53920, 10/0 MB/s
INFO: status: 54% (593738137600/1099511627776), sparse 52% (582737952768), duration 54933, 10/0 MB/s
INFO: status: 55% (604741566464/1099511627776), sparse 53% (593590673408), duration 55948, 10/0 MB/s
INFO: status: 56% (615733526528/1099511627776), sparse 54% (604366962688), duration 56960, 10/0 MB/s
INFO: status: 57% (626726141952/1099511627776), sparse 55% (615316996096), duration 57973, 10/0 MB/s
INFO: status: 58% (637725442048/1099511627776), sparse 56% (626314969088), duration 58994, 10/0 MB/s
INFO: status: 59% (648714584064/1099511627776), sparse 57% (637304090624), duration 60007, 10/0 MB/s
INFO: status: 60% (659716767744/1099511627776), sparse 58% (648306249728), duration 61019, 10/0 MB/s
INFO: status: 61% (670705057792/1099511627776), sparse 59% (659294519296), duration 62032, 10/0 MB/s
INFO: status: 62% (681701933056/1099511627776), sparse 60% (670291374080), duration 63045, 10/0 MB/s
INFO: status: 63% (692700119040/1099511627776), sparse 61% (681289539584), duration 64059, 10/0 MB/s
INFO: status: 64% (703691096064/1099511627776), sparse 62% (692280483840), duration 65074, 10/0 MB/s
INFO: status: 65% (714688823296/1099511627776), sparse 63% (703278190592), duration 66088, 10/0 MB/s
INFO: status: 66% (725681831936/1099511627776), sparse 64% (714271178752), duration 67101, 10/0 MB/s
INFO: status: 67% (736678641664/1099511627776), sparse 65% (725267959808), duration 68117, 10/0 MB/s
INFO: status: 68% (747677351936/1099511627776), sparse 66% (736266633216), duration 69131, 10/0 MB/s
INFO: status: 69% (758670426112/1099511627776), sparse 67% (747259686912), duration 70144, 10/0 MB/s
INFO: status: 70% (769663631360/1099511627776), sparse 68% (758252871680), duration 71157, 10/0 MB/s
INFO: status: 71% (780658343936/1099511627776), sparse 69% (769247535104), duration 72170, 10/0 MB/s
INFO: status: 72% (791649124352/1099511627776), sparse 70% (780238295040), duration 73182, 10/0 MB/s
INFO: status: 73% (802653405184/1099511627776), sparse 71% (791242555392), duration 74196, 10/0 MB/s
INFO: status: 74% (813643661312/1099511627776), sparse 72% (802232750080), duration 75208, 10/0 MB/s
INFO: status: 75% (824643551232/1099511627776), sparse 73% (813232619520), duration 76221, 10/0 MB/s
INFO: status: 76% (835637542912/1099511627776), sparse 74% (824226590720), duration 77236, 10/0 MB/s
INFO: status: 77% (846632255488/1099511627776), sparse 75% (835221278720), duration 78249, 10/0 MB/s
INFO: status: 78% (857620742144/1099511627776), sparse 76% (846209728512), duration 79261, 10/0 MB/s
INFO: status: 79% (868616372224/1099511627776), sparse 77% (857205338112), duration 80276, 10/0 MB/s
INFO: status: 80% (879614033920/1099511627776), sparse 78% (868202979328), duration 81296, 10/0 MB/s
INFO: status: 81% (890612678656/1099511627776), sparse 79% (879201333248), duration 82308, 10/0 MB/s
INFO: status: 82% (901599723520/1099511627776), sparse 80% (890188357632), duration 83321, 10/0 MB/s
INFO: status: 83% (912597778432/1099511627776), sparse 81% (901186392064), duration 84334, 10/0 MB/s
INFO: status: 84% (923599896576/1099511627776), sparse 82% (912188489728), duration 85348, 10/0 MB/s
INFO: status: 85% (934586220544/1099511627776), sparse 83% (922319630336), duration 86364, 10/0 MB/s
INFO: status: 86% (945584275456/1099511627776), sparse 84% (932231401472), duration 87380, 10/1 MB/s
INFO: status: 87% (956584820736/1099511627776), sparse 85% (943231926272), duration 88393, 10/0 MB/s
INFO: status: 88% (967580188672/1099511627776), sparse 86% (954227261440), duration 89407, 10/0 MB/s
INFO: status: 89% (978567168000/1099511627776), sparse 87% (965214220288), duration 90418, 10/0 MB/s
INFO: status: 90% (989567320064/1099511627776), sparse 88% (976019005440), duration 91431, 10/0 MB/s
INFO: status: 91% (1000563408896/1099511627776), sparse 89% (986832576512), duration 92445, 10/0 MB/s
INFO: status: 92% (1011556286464/1099511627776), sparse 90% (997825433600), duration 93457, 10/0 MB/s
INFO: status: 93% (1022550278144/1099511627776), sparse 91% (1008819400704), duration 94469, 10/0 MB/s
INFO: status: 94% (1033541648384/1099511627776), sparse 92% (1019810750464), duration 95483, 10/0 MB/s
INFO: status: 95% (1044541407232/1099511627776), sparse 93% (1030810480640), duration 96497, 10/0 MB/s
INFO: status: 96% (1055536709632/1099511627776), sparse 94% (1041805762560), duration 97510, 10/0 MB/s
INFO: status: 97% (1066533453824/1099511627776), sparse 95% (1052802469888), duration 98523, 10/0 MB/s
INFO: status: 98% (1077523251200/1099511627776), sparse 96% (1063792246784), duration 99535, 10/0 MB/s
INFO: status: 99% (1088520257536/1099511627776), sparse 97% (1074789232640), duration 100548, 10/0 MB/s
INFO: status: 100% (1099511627776/1099511627776), sparse 98% (1085691920384), duration 101564, 10/0 MB/s
INFO: transferred 1099511 MB in 101564 seconds (10 MB/s)
INFO: archive file size: 6.81GB
INFO: Finished Backup of VM 113 (28:12:45)
INFO: Backup job finished successfully
TASK OK

ANY suggestion would be helpfull cause im seriously fucked right now..

EDIT: I have not configured the rest of my system to use jumbo frames.. so the switch configuration was wrong..

Thanks

Casper

 

[offerlam]

OK..

So I have been testing and testing and my suspicion encrease that its a proxmox og debian problem... not my switch cabling, switch setup or nas setup.. I have created a case with synology just to be sure...

but i also thought maybe it was my lacp configuration.. so i started looking into how that works in Linux/Debian i figured maybe i had done something wrong in this department..

i found this command that shows the status of the bond

/proc/net/bonding/bond0

root@proxmox01:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

802.3ad info
LACP rate: slow
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 1
        Actor Key: 17
        Partner Key: 14
        Partner Mac Address: d4:c9:ef:3a:fe:e0

Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e8:39:35:b7:c7:4c
Aggregator ID: 1
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: e8:39:35:b7:c7:4d
Aggregator ID: 2
Slave queue ID: 0

from my perspektive this looks ok and confirms that my interfaces have negotiated the correct speed of 1Gb

this is my interface file from one of the nodes.. hope anyone can maybe see and issue.. cause im getting really desperate here..

root@proxmox01:~# cat /etc/network/interfaces
# network interface settings
auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

iface eth3 inet manual

auto bond0
iface bond0 inet manual
        slaves eth2 eth3
        bond_miimon 100
        bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
        address  10.10.99.21
        netmask  255.255.255.0
        gateway  10.10.99.1
        bridge_ports bond0
        bridge_stp off
        bridge_fd 0
        bridge_maxage 0
        bridge_ageing 0
        bridge_maxwait 0

auto vmbr1
iface vmbr1 inet manual
        bridge_ports bond0.2
        bridge_stp off
        bridge_fd 0
        bridge_maxage 0
        bridge_ageing 0
        bridge_maxwait 0

Hope someone can help! or atleast give an suggestion

Thanks

Casper

 

[gkovacs]

I might have missed something, but you seem to be complaining of your backups being write speed limited (be it to a local disk or a network folder), while there is no information presented about the read speed, or IO performance of your disk subsystem holding your virtual machines.

Read speed limited backups
Your reported 66 Mbytes / sec would be an absolutely decent speed for vzdump when it has to random read, tar and zip many small files from spinning disk based arrays (like when you are backing up OpenVZ guests), not to mention the overhead from the LVM snapshot and concurrent disk IO.

On our 6 and 8-disk Adaptec RAID10 arrays OpenVZ guests get backed up at an average of 20-30 MBytes per second (although there are guests with many small files which stay well under 10 MBytes / sec), so backups in this case are read speed limited.

Write speed limited backups
KVM guests are different, as they present one big sequential read for vzdump which - in theory - should be much faster (high concurrent IO can still affect it negatively), so you can expect network or NAS write performance to limit your backup speed.

Our KVM machine backups average 50-100 Mbytes / sec, but could reach 300-400 MBytes / sec (I just tested this).
BTW we limit this to 48 MBytes / sec in vzdump.conf to maintain low iowait on our nodes.

Also don't forget: your 66 MBytes / sec is equal to 528 Mbits / sec, or half the capacity of a gigabit link. Vzdump reports in MegaBytes per second, network speed is in Megabits per second (x8).

The reason our KVM backups to gigabit NFS can reach 300-400 MBytes per second is compression:

[COLOR=#000000][FONT=tahoma]INFO: transferred 21474 MB in 64 seconds (335 MB/s)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]INFO: archive file size: 722MB[/FONT][/COLOR]

In this case vzdump backed up a 21 GigaByte RAW KVM to a 722 MegaByte file (30:1 compression), reading from disk at 335 MByes / sec, but only used about 90 Megabits / sec network IO (722 MBytes x 8 / 64 seconds).

What about your system
So to better analyze your problem, tell us about your disk subsystem holding your VMs:
- are you running single disk, mdraid of hardware RAID?
- how many disks or SSDs, and what kind?
- what type of storage / RAID controller?
- what is your RAID level?
- are you using ext3 or ext4?
- what IO scheduler you have configured?
- do you have relatime or noatime set?
- what does pveperf say for buffered reads during no IO load?

Also, try to backup a big but almost empty RAW disk KVM, and include your vzdump log.

 

[offerlam]

Hi gkovacs,

and thanks .. no THANK YOU for offering your insite into proxmox backup behavior!!

To set some facts on the table..

my setup is this

3 * HP 165 gen7 servers. Two network attached 1gb cable to each network switch in a LACP configuration
2 * HP 1810-g24 v2 switche connected with 1gb trunk to eachother
2* DS1513+ synology NAS each connected with two 1gb cables to each network switch in a LACP configuration..

more details on the DS1513+

one is for backup and one is for all the other stuff, vms templates and so on..

backup nas has 2 * 4TB 7200 rpm WD4000f9yz in a raid one configuration - this is software raid
storage nas has 3 * 4TB 72000 rpm WD4000f8yz in a raid 5 configuration - this is software raid

---

the ONLY way i get 66 MBs is when i have jumbo frames set. without Jumbo Frames my installation performs with 10MBs when doing backups. There are log post futher up to dokument this. So if you think 66 should be a good speed I only get this with jumbo frames..

ALSO I get the same backup speeds when i do backup to local disk

all my VMs are KVM

---

as for your questions:

- are you running single disk, mdraid of hardware RAID?
backup nas is raid 1 and storage nas is raid 5 - both software since its synology nas

- how many disks or SSDs, and what kind?
2 disks in backup and 3 in storage - no ssd

- what type of storage / RAID controller?
software raid controller on both NAS the NAS are Synology DS1513+

- what is your RAID level?
backup is raid 1 and storage is raid 5

- are you using ext3 or ext4?
ext4

- what IO scheduler you have configured?
I don't know what that is, but i suspect that my nas don't have any

- do you have relatime or noatime set?
I don't know what that is?

- what does pveperf say for buffered reads during no IO load?

pveperf for backup nfs share

root@proxmox02:~# pveperf /mnt/pve/Backup01
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      633521
HD SIZE:           3663.09 GB (10.10.99.11:/volume1/Backup01)
FSYNCS/SECOND:     312.62
DNS EXT:           62.94 ms
DNS INT:           35.42 ms (ins.local)
root@proxmox02:~# pveperf /mnt/pve/Backup01
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      644965
HD SIZE:           3663.09 GB (10.10.99.11:/volume1/Backup01)
FSYNCS/SECOND:     411.96
DNS EXT:           62.00 ms
DNS INT:           39.70 ms (ins.local)
root@proxmox02:~# pveperf /mnt/pve/Backup01
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      640412
HD SIZE:           3663.09 GB (10.10.99.11:/volume1/Backup01)
FSYNCS/SECOND:     298.80
DNS EXT:           51.96 ms
DNS INT:           51.31 ms (ins.local)

pveperf for storage nfs share for VMs

root@proxmox02:~# pveperf /mnt/pve/Storage01_Vms/
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      650768
HD SIZE:           7384.35 GB (10.10.99.10:/volume1/VMs)
FSYNCS/SECOND:     1295.15
DNS EXT:           52.84 ms
DNS INT:           35.38 ms (ins.local)
root@proxmox02:~# pveperf /mnt/pve/Storage01_Vms/
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      636337
HD SIZE:           7384.35 GB (10.10.99.10:/volume1/VMs)
FSYNCS/SECOND:     1307.45
DNS EXT:           59.76 ms
DNS INT:           37.98 ms (ins.local)
root@proxmox02:~# pveperf /mnt/pve/Storage01_Vms/
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      569644
HD SIZE:           7384.35 GB (10.10.99.10:/volume1/VMs)
FSYNCS/SECOND:     1306.63
DNS EXT:           49.24 ms
DNS INT:           31.36 ms (ins.local)

and just your plan old pveperf

root@proxmox02:~# pveperf
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      622521
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    159.24 MB/sec
AVERAGE SEEK TIME: 4.22 ms
FSYNCS/SECOND:     59.02
DNS EXT:           58.85 ms
DNS INT:           30.80 ms (ins.local)
root@proxmox02:~# pveperf
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      643331
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    154.84 MB/sec
AVERAGE SEEK TIME: 4.17 ms
FSYNCS/SECOND:     61.91
DNS EXT:           64.83 ms
DNS INT:           43.69 ms (ins.local)
root@proxmox02:~# pveperf
CPU BOGOMIPS:      67196.96
REGEX/SECOND:      597630
HD SIZE:           94.49 GB (/dev/mapper/pve-root)
BUFFERED READS:    165.13 MB/sec
AVERAGE SEEK TIME: 4.18 ms
FSYNCS/SECOND:     63.02
DNS EXT:           62.11 ms
DNS INT:           35.35 ms (ins.local)

there are more pveperf data in one of my post further up aswell

---

Here are some of the things im still looking into..

1.

I'm assuming i would get more than 10MBs or 66MBs (with jumbo frames). but based on the fact i get the same speeds i would assume my problem MAY be with the storage nas.. and not the backup

2.

The way i found out my 66MBs was because of jumbo frames was when i were going through the switches and saw that one had ticked on jumbo frames. when i removed this my backup speed dropped to 10MBs. Since my servers run 1gbs to the switch and likewise for the NAS AND the fact that when i turn on or off a technology on the switch that increase performance and i also see a performance de/increase I could assume I have a switch problem? If it was a server or NAS problem it shouldn't help turning on jumbo frames right? by the way i tried implementing jumbo frames on the servers and NAS aswell but backup speed didn't increase.. I will make a case with HP about this

3.

eth 3 and 2 are the ports connected from the servers to the switch. eth4 right diode on the server is red all the time no blinking or anything.. I need to find a way to see if there is a error no the interface. It does this on all servers so im thinking its because im running LACP and that shows me that eth3 is on standby and not active? I will make a case with HP about this

---

As for backup of a RAW vm with lots of harddisk space. I made a KVM ubuntu 64 bit LVM with 1024GB of disk. I won't post the entire log now since I still have 66MBs so it made no difference. I will post the entire log when its done in like 4 or 5 hours

INFO: starting new backup job: vzdump 103 --remove 0 --mode snapshot --compress lzo --storage Backup01 --node proxmox00
INFO: Starting Backup of VM 103 (qemu)
INFO: status = running
INFO: update VM 103: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backup01/dump/vzdump-qemu-103-2014_02_17-09_31_51.vma.lzo'
INFO: started backup task 'bdcb0c06-04e0-4c3d-807e-d3bb9e223a37'
INFO: status: 0% (198246400/1099511627776), sparse 0% (154861568), duration 3, 66/14 MB/s
INFO: status: 1% (11009064960/1099511627776), sparse 0% (10259406848), duration 167, 65/4 MB/s
INFO: status: 2% (22046441472/1099511627776), sparse 1% (21296095232), duration 333, 66/0 MB/s
INFO: status: 3% (33031520256/1099511627776), sparse 2% (32280817664), duration 499, 66/0 MB/s

Again THANKS for pitching in!

Casper

 

[gkovacs]

Since both your VM storage and your backup storage are on the same network, there is not much point in test backing up a huge empty RAW disk VM - that would only show the read speed of local storage. Compression isn't going to help much in this case either.

As you are using the same LAN to both read from VM storage and write to backup storage during backups, you are essentially limited to half your LAN capacity, which is around 64 MBytes / sec (512 Mbits / sec) for half of a gigabit link. Your link aggregation does not seem to be working.

I would probably try separating storages to different (V)LANs, so that both directions could get at least a full gigabit link.

 

[offerlam]

I don't understand why that's a problem?

do you mean it happens because the data read data and write data is send fromt the same server nic?

here is the result of my backup job with the raw image.. i know you said it didn't matter now but now you get it

INFO: starting new backup job: vzdump 103 --remove 0 --mode snapshot --compress lzo --storage Backup01 --node proxmox00
INFO: Starting Backup of VM 103 (qemu)
INFO: status = running
INFO: update VM 103: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/Backup01/dump/vzdump-qemu-103-2014_02_17-09_31_51.vma.lzo'
INFO: started backup task 'bdcb0c06-04e0-4c3d-807e-d3bb9e223a37'
INFO: status: 0% (198246400/1099511627776), sparse 0% (154861568), duration 3, 66/14 MB/s
INFO: status: 1% (11009064960/1099511627776), sparse 0% (10259406848), duration 167, 65/4 MB/s
INFO: status: 2% (22046441472/1099511627776), sparse 1% (21296095232), duration 333, 66/0 MB/s
INFO: status: 3% (33031520256/1099511627776), sparse 2% (32280817664), duration 499, 66/0 MB/s
INFO: status: 4% (43981340672/1099511627776), sparse 3% (43230617600), duration 663, 66/0 MB/s
INFO: status: 5% (55028547584/1099511627776), sparse 4% (54277533696), duration 829, 66/0 MB/s
INFO: status: 6% (65973518336/1099511627776), sparse 5% (65222483968), duration 995, 65/0 MB/s
INFO: status: 7% (76991102976/1099511627776), sparse 6% (76203859968), duration 1161, 66/0 MB/s
INFO: status: 8% (88002002944/1099511627776), sparse 7% (87214469120), duration 1324, 67/0 MB/s
INFO: status: 9% (98964144128/1099511627776), sparse 8% (98176319488), duration 1489, 66/0 MB/s
INFO: status: 10% (109999816704/1099511627776), sparse 9% (109211967488), duration 1655, 66/0 MB/s
INFO: status: 11% (120949178368/1099511627776), sparse 10% (120161308672), duration 1818, 67/0 MB/s
INFO: status: 12% (131941793792/1099511627776), sparse 11% (131153891328), duration 1986, 65/0 MB/s
INFO: status: 13% (142998110208/1099511627776), sparse 12% (142210121728), duration 2155, 65/0 MB/s
INFO: status: 14% (153943801856/1099511627776), sparse 13% (153155792896), duration 2320, 66/0 MB/s
INFO: status: 15% (164976918528/1099511627776), sparse 14% (164188876800), duration 2486, 66/0 MB/s
INFO: status: 16% (175969402880/1099511627776), sparse 15% (175181328384), duration 2649, 67/0 MB/s
INFO: status: 17% (186966016000/1099511627776), sparse 16% (186177200128), duration 2814, 66/0 MB/s
INFO: status: 18% (197969641472/1099511627776), sparse 17% (197180801024), duration 2977, 67/0 MB/s
INFO: status: 19% (208929619968/1099511627776), sparse 18% (208140759040), duration 3140, 67/0 MB/s
INFO: status: 20% (219966930944/1099511627776), sparse 19% (219177984000), duration 3305, 66/0 MB/s
INFO: status: 21% (230926909440/1099511627776), sparse 20% (230137942016), duration 3468, 67/0 MB/s
INFO: status: 22% (241954979840/1099511627776), sparse 21% (241165991936), duration 3632, 67/0 MB/s
INFO: status: 23% (252936978432/1099511627776), sparse 22% (252147970048), duration 3799, 65/0 MB/s
INFO: status: 24% (263907966976/1099511627776), sparse 23% (263118938112), duration 3963, 66/0 MB/s
INFO: status: 25% (274892390400/1099511627776), sparse 24% (274103341056), duration 4125, 67/0 MB/s
INFO: status: 26% (285912072192/1099511627776), sparse 25% (285122932736), duration 4292, 65/0 MB/s
INFO: status: 27% (296870477824/1099511627776), sparse 26% (296081022976), duration 4456, 66/0 MB/s
INFO: status: 28% (307880001536/1099511627776), sparse 27% (307090526208), duration 4619, 67/0 MB/s
INFO: status: 29% (318919344128/1099511627776), sparse 28% (318129848320), duration 4786, 66/0 MB/s
INFO: status: 30% (329892823040/1099511627776), sparse 29% (329103036416), duration 4953, 65/0 MB/s
INFO: status: 31% (340874297344/1099511627776), sparse 30% (340084490240), duration 5119, 66/0 MB/s
INFO: status: 32% (351888605184/1099511627776), sparse 31% (351098712064), duration 5287, 65/0 MB/s
INFO: status: 33% (362849566720/1099511627776), sparse 32% (362059653120), duration 5451, 66/0 MB/s
INFO: status: 34% (373849587712/1099511627776), sparse 33% (373059653632), duration 5616, 66/0 MB/s
INFO: status: 35% (384855506944/1099511627776), sparse 34% (384065511424), duration 5780, 67/0 MB/s
INFO: status: 36% (395837571072/1099511627776), sparse 35% (395047489536), duration 5951, 64/0 MB/s
INFO: status: 37% (406831431680/1099511627776), sparse 36% (406041329664), duration 6117, 66/0 MB/s
INFO: status: 38% (417858846720/1099511627776), sparse 37% (417068711936), duration 6282, 66/0 MB/s
INFO: status: 39% (428850479104/1099511627776), sparse 38% (428059987968), duration 6443, 68/0 MB/s
INFO: status: 40% (439850893312/1099511627776), sparse 39% (439060369408), duration 6603, 68/0 MB/s
INFO: status: 41% (450833481728/1099511627776), sparse 40% (450042937344), duration 6764, 68/0 MB/s
INFO: status: 42% (461817839616/1099511627776), sparse 41% (460897050624), duration 6929, 66/0 MB/s
INFO: status: 43% (472792236032/1099511627776), sparse 42% (471808286720), duration 7096, 65/0 MB/s
INFO: status: 44% (483818995712/1099511627776), sparse 43% (482752696320), duration 7258, 68/0 MB/s
INFO: status: 45% (494791557120/1099511627776), sparse 44% (493725237248), duration 7423, 66/0 MB/s
INFO: status: 46% (505813860352/1099511627776), sparse 45% (504747520000), duration 7589, 66/0 MB/s
INFO: status: 47% (516803919872/1099511627776), sparse 46% (515737559040), duration 7756, 65/0 MB/s
INFO: status: 48% (527788015616/1099511627776), sparse 47% (526721634304), duration 7922, 66/0 MB/s
INFO: status: 49% (538771587072/1099511627776), sparse 48% (537705119744), duration 8090, 65/0 MB/s
INFO: status: 50% (549799854080/1099511627776), sparse 49% (548599955456), duration 8257, 66/0 MB/s
INFO: status: 51% (560800530432/1099511627776), sparse 50% (559600607232), duration 8423, 66/0 MB/s
INFO: status: 52% (571807236096/1099511627776), sparse 51% (570607292416), duration 8587, 67/0 MB/s
INFO: status: 53% (582742573056/1099511627776), sparse 52% (581542596608), duration 8752, 66/0 MB/s
INFO: status: 54% (593749016576/1099511627776), sparse 53% (592549019648), duration 8913, 68/0 MB/s
INFO: status: 55% (604774465536/1099511627776), sparse 54% (603574448128), duration 9077, 67/0 MB/s
INFO: status: 56% (615750762496/1099511627776), sparse 55% (614550724608), duration 9238, 68/0 MB/s
INFO: status: 57% (626749014016/1099511627776), sparse 56% (625548955648), duration 9399, 68/0 MB/s
INFO: status: 58% (637762273280/1099511627776), sparse 57% (636562194432), duration 9564, 66/0 MB/s
INFO: status: 59% (648756920320/1099511627776), sparse 58% (647556820992), duration 9727, 67/0 MB/s
INFO: status: 60% (659761463296/1099511627776), sparse 59% (658561339392), duration 9890, 67/0 MB/s
INFO: status: 61% (670705385472/1099511627776), sparse 60% (669505241088), duration 10054, 66/0 MB/s
INFO: status: 62% (681701998592/1099511627776), sparse 61% (680501833728), duration 10222, 65/0 MB/s
INFO: status: 63% (692733673472/1099511627776), sparse 62% (691533475840), duration 10383, 68/0 MB/s
INFO: status: 64% (703738544128/1099511627776), sparse 63% (702538326016), duration 10544, 68/0 MB/s
INFO: status: 65% (714719035392/1099511627776), sparse 64% (713518796800), duration 10706, 67/0 MB/s
INFO: status: 66% (725723578368/1099511627776), sparse 65% (724523319296), duration 10870, 67/0 MB/s
INFO: status: 67% (736713441280/1099511627776), sparse 66% (735513149440), duration 11035, 66/0 MB/s
INFO: status: 68% (747671060480/1099511627776), sparse 67% (746470744064), duration 11200, 66/0 MB/s
INFO: status: 69% (758725476352/1099511627776), sparse 68% (757525139456), duration 11368, 65/0 MB/s
INFO: status: 70% (769671888896/1099511627776), sparse 69% (768471531520), duration 11530, 67/0 MB/s
INFO: status: 71% (780678987776/1099511627776), sparse 70% (779478609920), duration 11693, 67/0 MB/s
INFO: status: 72% (791661707264/1099511627776), sparse 71% (790461308928), duration 11851, 69/0 MB/s
INFO: status: 73% (802669330432/1099511627776), sparse 72% (801468911616), duration 12016, 66/0 MB/s
INFO: status: 74% (813689995264/1099511627776), sparse 73% (812489555968), duration 12180, 67/0 MB/s
INFO: status: 75% (824636211200/1099511627776), sparse 74% (823435751424), duration 12344, 66/0 MB/s
INFO: status: 76% (835678502912/1099511627776), sparse 75% (834478018560), duration 12512, 65/0 MB/s
INFO: status: 77% (846648770560/1099511627776), sparse 76% (845448265728), duration 12677, 66/0 MB/s
INFO: status: 78% (857680052224/1099511627776), sparse 77% (856479526912), duration 12844, 66/0 MB/s
INFO: status: 79% (868676861952/1099511627776), sparse 78% (867476316160), duration 13005, 68/0 MB/s
INFO: status: 80% (879633498112/1099511627776), sparse 79% (878428987392), duration 13170, 66/0 MB/s
INFO: status: 81% (890672250880/1099511627776), sparse 80% (889467449344), duration 13335, 66/0 MB/s
INFO: status: 82% (901653397504/1099511627776), sparse 81% (900448575488), duration 13502, 65/0 MB/s
INFO: status: 83% (912602759168/1099511627776), sparse 82% (911397900288), duration 13665, 67/0 MB/s
INFO: status: 84% (923623358464/1099511627776), sparse 83% (922418479104), duration 13831, 66/0 MB/s
INFO: status: 85% (934599983104/1099511627776), sparse 84% (933395079168), duration 13999, 65/0 MB/s
INFO: status: 86% (945592074240/1099511627776), sparse 85% (944254038016), duration 14164, 66/0 MB/s
INFO: status: 87% (956577677312/1099511627776), sparse 86% (955239620608), duration 14330, 66/0 MB/s
INFO: status: 88% (967587004416/1099511627776), sparse 87% (966248927232), duration 14495, 66/0 MB/s
INFO: status: 89% (978581389312/1099511627776), sparse 88% (977243291648), duration 14658, 67/0 MB/s
INFO: status: 90% (989602840576/1099511627776), sparse 89% (988260290560), duration 14816, 69/0 MB/s
INFO: status: 91% (1000590213120/1099511627776), sparse 90% (999247601664), duration 14982, 66/0 MB/s
INFO: status: 92% (1011563364352/1099511627776), sparse 91% (1010220720128), duration 15148, 66/0 MB/s
INFO: status: 93% (1022583111680/1099511627776), sparse 92% (1021240442880), duration 15316, 65/0 MB/s
INFO: status: 94% (1033576251392/1099511627776), sparse 93% (1032233562112), duration 15479, 67/0 MB/s
INFO: status: 95% (1044543897600/1099511627776), sparse 94% (1043201187840), duration 15640, 68/0 MB/s
INFO: status: 96% (1055540969472/1099511627776), sparse 95% (1054198239232), duration 15801, 68/0 MB/s
INFO: status: 97% (1066587455488/1099511627776), sparse 96% (1065187536896), duration 15967, 66/0 MB/s
INFO: status: 98% (1077572075520/1099511627776), sparse 97% (1076134477824), duration 16130, 67/0 MB/s
INFO: status: 99% (1088523599872/1099511627776), sparse 98% (1087085981696), duration 16296, 65/0 MB/s
INFO: status: 100% (1099511627776/1099511627776), sparse 99% (1098073694208), duration 16461, 66/0 MB/s
INFO: transferred 1099511 MB in 16461 seconds (66 MB/s)
INFO: archive file size: 666MB
INFO: Finished Backup of VM 103 (04:34:23)
INFO: Backup job finished successfully
TASK OK

 

[gkovacs]

I don't understand why that's a problem?

do you mean it happens because the data read data and write data is send fromt the same server nic?

Yes, that's exactly what I mean.

You have a single gigabit link that you use for both reading and writing at the same time. A single gigabit link is 128 MB/s, you are getting half of that for your backup speed, since you are using the link in both directions at the same time.

In theory gigabit links should be full duplex (full speed in both directions), but in this case your link is not working like that. Maybe this is caused by your link aggregation (LACP), which is also clearly not working.

 

[offerlam]

Yes, that's exactly what I mean.

You have a single gigabit link that you use for both reading and writing at the same time. A single gigabit link is 128 MB/s, you are getting half of that for your backup speed, since you are using the link in both directions at the same time.

In theory gigabit links should be full duplex (full speed in both directions), but in this case your link is not working like that. Maybe this is caused by your link aggregation (LACP), which is also clearly not working.

As for the link aggregation not working..

eth 2 and 3 are the ones reserved for the LACP trunk. they are connect to each switch.. as I understood it from LACP doing it this way means one link will be down and the other will be up .. and if the switch where the interface is up goes down, maybe power failure or something, the second link will come alive and continue on.. isn't that right? and if i am right that mean i only have 1gb link avalible ..that being said I should still get 1gb full duplex as you say which i clearly don't...but im not sure my LACP is broken.. its just because each link is connected to each switch and not a single switch..

do you agree?

 

[offerlam]

Hi all,

Ok so i have been on and off this problem now.. time caught up to me so i couldn't give it the time it needed... but I went and installed iperf on my backup NAS and on my servers and here are the results..

Backup01> iperf -c 10.10.99.22
------------------------------------------------------------
Client connecting to 10.10.99.22, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[ 5] local 10.10.99.11 port 48657 connected with 10.10.99.22 port 5001
write2 failed: Interrupted system call
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
Backup01> iperf -c 10.10.99.20
------------------------------------------------------------
Client connecting to 10.10.99.20, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[ 5] local 10.10.99.11 port 46255 connected with 10.10.99.20 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.10 GBytes 942 Mbits/sec
Backup01> iperf -c 10.10.99.21
------------------------------------------------------------
Client connecting to 10.10.99.21, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[ 5] local 10.10.99.11 port 42338 connected with 10.10.99.21 port 5001
write2 failed: Interrupted system call
[ ID] Interval Transfer Bandwidth
[ 5] 0.0-10.0 sec 1.10 GBytes 943 Mbits/sec
Backup01>

gkovacs im pretty new to IPERF.. but doesn't this show my full duplex is working?

I can't disprove that part of my LACP is NOT working.. but i can say that if i unplug a cable from one of the nodes or pull the power from one of the two switch fail over works perfectly.. with a max of 1 packet loss

Also HP 3ed level told me that this would work due to the way LACP protocol was working..

I have also fully updated my 3 proxmox nodes