Backup very slow on Proxmox 3

[wahmed]

I just took a look at some documentation. Did you see this:
http://pve.proxmox.com/wiki/Performance_Tweaks

I actually read that documentation quite a few times. Thats where i got to learn about different cache in Proxmox.
So far i tried about 8 different backups, all with consistent results. Backup speed around 105 mib/s with Writethrough, Writeback cache. But according to the documentation, writethrough is safer but slow. Apparently in my case it is the opposite. I thought about hardware issue, but backup "is" happening at gigabit speed, just with different settings.
..................
Just to try out something different, i moved one of the VM on local disk, then did a backup. The speed passed 300 mib/s!!! Two different backup, result is same. Cache option really did not matter. So obviously, Proxmox is doing its job.
What is going on then ? Why is backup slowing down over network without cache on raw?

 

[mir]

If your setting is cache=none then all caching is left to the client but I cannot think of any reason that this should affect backup speed since making a backup solely concerns writes outside the client - the client is not aware of a backup is in progress. The only explanation I can give is that you are doing an online backup with a lot of writes happening at the same time inside the client an given cache=none means the client needs to wait for commit of the write which at the same time will put locks on the file which constitutes the file which is currently being backed up. Locks on this file could inflict the backup session resulting in slowing down the data transfer.

 

[wahmed]

If your setting is cache=none then all caching is left to the client but I cannot think of any reason that this should affect backup speed since making a backup solely concerns writes outside the client - the client is not aware of a backup is in progress. ........

Thats exactly my thinking!! During all the backup test the VM has not been running but powered off. So VM guest side cache does not even come into effect. This is one odd mystery.

 

[BitRausch]

ok symmcom. I am out of ideas. I am not an expert on this subject.
Here some things I would try/check:
- remove the storage and create a new storage (just in pve. Not deleting the backup)
- check network parameters including hosts, jumbo frames (if set), dns server entry...
- check cpu load during backup
- check HDD health
- use ntop to monitor the transfer and see if there is a ongoing stream of data or does the transfer start/stop
- Is your system using swap during backup ?
- On ZFS there is a parameter called nbmand Quote "Nbmand is used for system-wide filelocking coordination between SMB and NFS"
Is this on or off ?

I use Intel NIC's for my network. In my experience Realtek or Broadcom do not perform as good as intel NIC's.

PS: More Information on nbmand is here.

 

[BitRausch]

Thats exactly my thinking!! During all the backup test the VM has not been running but powered off. So VM guest side cache does not even come into effect. This is one odd mystery.

Are you sure about this ? In Version 3 when you do a backup in Snapshot mode and the VM is stopped the VM is put into a pre-start mode. That is the VM starts but does not boot. I think it also allocates the memory...

 

[jmjosebest]

Also I detect that backups are more slow (biggest dump file size) with Proxmox v3.
And backups takes more disk space.


With Proxmox v2:

INFO: starting new backup job: vzdump --quiet 1 --mailto xx@xx.com --mode snapshot --compress lzo --storage nfsabs1-private --all 1 --exclude 112
INFO: Starting Backup of VM 151 (qemu)
INFO: status = running
INFO: exclude disk 'virtio1' (backup=no)
INFO: backup mode: snapshot
INFO: bandwidth limit: 20480 KB/s
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-151-2013_06_06-00_05_02.vma.lzo'
INFO: started backup task '9e4e8d02-c9cb-4d77-90ce-958ea3d22312'
INFO: status: 0% (93257728/450971566080), sparse 0% (28561408), duration 3, 31/21 MB/s
INFO: status: 1% (4513202176/450971566080), sparse 0% (109432832), duration 241, 18/18 MB/s
INFO: status: 2% (9030729728/450971566080), sparse 0% (317718528), duration 469, 19/18 MB/s
INFO: status: 3% (13549240320/450971566080), sparse 0% (394031104), duration 709, 18/18 MB/s
INFO: status: 4% (18051104768/450971566080), sparse 0% (422408192), duration 934, 20/19 MB/s
INFO: status: 5% (22568370176/450971566080), sparse 0% (467628032), duration 1160, 19/19 MB/s
INFO: status: 6% (27072331776/450971566080), sparse 0% (495230976), duration 1374, 21/20 MB/s
INFO: status: 7% (31584878592/450971566080), sparse 0% (533217280), duration 1584, 21/21 MB/s
INFO: status: 8% (36080582656/450971566080), sparse 0% (641196032), duration 1777, 23/22 MB/s
INFO: status: 9% (40610168832/450971566080), sparse 0% (686170112), duration 2007, 19/19 MB/s
INFO: status: 10% (45102530560/450971566080), sparse 0% (701898752), duration 2216, 21/21 MB/s
INFO: status: 90% (405884370944/450971566080), sparse 4% (18700648448), duration 20379, 20/19 MB/s
INFO: status: 91% (410389839872/450971566080), sparse 4% (18823057408), duration 20602, 20/19 MB/s
INFO: status: 92% (414911823872/450971566080), sparse 4% (20975902720), duration 20822, 20/10 MB/s
INFO: status: 93% (419405824000/450971566080), sparse 4% (21234049024), duration 21041, 20/19 MB/s
INFO: status: 94% (423915880448/450971566080), sparse 4% (21375647744), duration 21275, 19/18 MB/s
INFO: status: 95% (428426395648/450971566080), sparse 4% (21616984064), duration 21521, 18/17 MB/s
INFO: status: 96% (432948838400/450971566080), sparse 4% (21766176768), duration 21746, 20/19 MB/s
INFO: status: 97% (437461843968/450971566080), sparse 4% (22087675904), duration 21972, 19/18 MB/s
INFO: status: 98% (441969147904/450971566080), sparse 4% (22232920064), duration 22229, 17/16 MB/s
INFO: status: 99% (446463541248/450971566080), sparse 4% (22379483136), duration 22464, 19/18 MB/s
INFO: status: 100% (450971566080/450971566080), sparse 5% (26681667584), duration 22674, 21/0 MB/s
INFO: transferred 450971 MB in 22674 seconds (19 MB/s)
INFO: archive file size: 218.46GB
INFO: delete old backup '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-151-2013_06_03-00_05_02.vma.lzo'
INFO: Finished Backup of VM 151 (06:18:05)
INFO: Starting Backup of VM 154 (qemu)
INFO: status = running
INFO: exclude disk 'virtio1' (backup=no)
INFO: backup mode: snapshot
INFO: bandwidth limit: 20480 KB/s
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-154-2013_06_06-06_23_29.vma.lzo'
INFO: started backup task '063f7835-fc92-4321-b8f1-8367740f973d'
INFO: status: 0% (69664768/450971566080), sparse 0% (23183360), duration 3, 23/15 MB/s
INFO: status: 1% (4529061888/450971566080), sparse 0% (1262321664), duration 210, 21/15 MB/s
INFO: status: 2% (9048948736/450971566080), sparse 0% (1617223680), duration 410, 22/20 MB/s
INFO: status: 3% (13549371392/450971566080), sparse 0% (1652330496), duration 599, 23/23 MB/s
INFO: status: 4% (18054905856/450971566080), sparse 0% (1743233024), duration 833, 19/18 MB/s
INFO: status: 5% (22554869760/450971566080), sparse 0% (1846775808), duration 1060, 19/19 MB/s
INFO: status: 6% (27070169088/450971566080), sparse 0% (1933447168), duration 1309, 18/17 MB/s
INFO: status: 7% (31577669632/450971566080), sparse 0% (2007887872), duration 1538, 19/19 MB/s
INFO: status: 8% (36085956608/450971566080), sparse 0% (2054160384), duration 1764, 19/19 MB/s
INFO: status: 9% (40598175744/450971566080), sparse 0% (2115911680), duration 1978, 21/20 MB/s
INFO: status: 10% (45113999360/450971566080), sparse 0% (2194812928), duration 2203, 20/19 MB/s
INFO: status: 90% (405888630784/450971566080), sparse 1% (8325271552), duration 19436, 21/20 MB/s
INFO: status: 91% (410401308672/450971566080), sparse 1% (8400625664), duration 19650, 21/20 MB/s
INFO: status: 92% (414895964160/450971566080), sparse 1% (8458285056), duration 19865, 20/20 MB/s
INFO: status: 93% (419422273536/450971566080), sparse 1% (8519675904), duration 20079, 21/20 MB/s
INFO: status: 94% (423922958336/450971566080), sparse 1% (8603471872), duration 20294, 20/20 MB/s
INFO: status: 95% (428437667840/450971566080), sparse 1% (8795422720), duration 20505, 21/20 MB/s
INFO: status: 96% (432948576256/450971566080), sparse 1% (8871096320), duration 20717, 21/20 MB/s
INFO: status: 97% (437455290368/450971566080), sparse 1% (9010835456), duration 20933, 20/20 MB/s
INFO: status: 98% (441953157120/450971566080), sparse 2% (9197699072), duration 21145, 21/20 MB/s
INFO: status: 99% (446464262144/450971566080), sparse 2% (9239441408), duration 21364, 20/20 MB/s
INFO: status: 100% (450971566080/450971566080), sparse 3% (13532692480), duration 21573, 21/1 MB/s
INFO: transferred 450971 MB in 21573 seconds (20 MB/s)
INFO: archive file size: 168.21GB
INFO: delete old backup '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-154-2013_06_03-06_18_39.vma.lzo'
INFO: Finished Backup of VM 154 (05:59:40)
INFO: Backup job finished successfully
TASK OK

With Proxmox v3:

INFO: starting new backup job: vzdump --quiet 1 --mailto xx@xx.com --mode snapshot --compress lzo --storage nfsabs1-private --all 1 --exclude 112,102
INFO: Starting Backup of VM 151 (qemu)
INFO: status = running
INFO: exclude disk 'virtio1' (backup=no)
INFO: backup mode: snapshot
INFO: bandwidth limit: 20480 KB/s
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-151-2013_06_11-00_05_02.vma.lzo'
INFO: started backup task '013885b0-936a-4701-a459-d5e00077c53a'
INFO: status: 0% (79495168/665719930880), sparse 0% (8482816), duration 3, 26/23 MB/s
INFO: status: 1% (6658785280/665719930880), sparse 0% (273625088), duration 316, 21/20 MB/s
INFO: status: 2% (13332905984/665719930880), sparse 0% (392970240), duration 649, 20/19 MB/s
INFO: status: 3% (19991494656/665719930880), sparse 0% (431144960), duration 989, 19/19 MB/s
INFO: status: 4% (26645561344/665719930880), sparse 0% (483274752), duration 1307, 20/20 MB/s
INFO: status: 5% (33296678912/665719930880), sparse 0% (666116096), duration 1607, 22/21 MB/s
INFO: status: 6% (39959789568/665719930880), sparse 0% (879910912), duration 1936, 20/19 MB/s
INFO: status: 7% (46610579456/665719930880), sparse 0% (910069760), duration 2258, 20/20 MB/s
INFO: status: 8% (53275197440/665719930880), sparse 0% (954646528), duration 2591, 20/19 MB/s
INFO: status: 9% (59918843904/665719930880), sparse 0% (1117347840), duration 2907, 21/20 MB/s
INFO: status: 10% (66581495808/665719930880), sparse 0% (1149763584), duration 3235, 20/20 MB/s
INFO: status: 90% (599162290176/665719930880), sparse 5% (36799320064), duration 30303, 16/15 MB/s
INFO: status: 91% (605815701504/665719930880), sparse 5% (37323153408), duration 30664, 18/16 MB/s
INFO: status: 92% (612463804416/665719930880), sparse 5% (37734174720), duration 31002, 19/18 MB/s
INFO: status: 93% (619138646016/665719930880), sparse 5% (38145183744), duration 31344, 19/18 MB/s
INFO: status: 94% (625796448256/665719930880), sparse 5% (38556192768), duration 31712, 18/16 MB/s
INFO: status: 95% (632449400832/665719930880), sparse 5% (38958829568), duration 32075, 18/17 MB/s
INFO: status: 96% (639099731968/665719930880), sparse 5% (39394979840), duration 32421, 19/17 MB/s
INFO: status: 97% (645754519552/665719930880), sparse 5% (39814373376), duration 32776, 18/17 MB/s
INFO: status: 98% (652407341056/665719930880), sparse 6% (40233762816), duration 33100, 20/19 MB/s
INFO: status: 99% (659067109376/665719930880), sparse 6% (40661532672), duration 33423, 20/19 MB/s
INFO: status: 100% (665719930880/665719930880), sparse 6% (46163107840), duration 33749, 20/3 MB/s
INFO: transferred 665719 MB in 33749 seconds (19 MB/s)
INFO: archive file size: 398.35GB
INFO: delete old backup '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-151-2013_06_08-00_05_02.vma.lzo'
INFO: Finished Backup of VM 151 (09:22:40)
INFO: Starting Backup of VM 154 (qemu)
INFO: status = running
INFO: exclude disk 'virtio1' (backup=no)
INFO: backup mode: snapshot
INFO: bandwidth limit: 20480 KB/s
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-154-2013_06_11-09_27_42.vma.lzo'
INFO: started backup task '1a30e8af-cb0d-42e2-b8c5-6627d4955df2'
INFO: status: 0% (111017984/665719930880), sparse 0% (7098368), duration 3, 37/34 MB/s
INFO: status: 1% (6660292608/665719930880), sparse 0% (1431502848), duration 272, 24/19 MB/s
INFO: status: 2% (13342081024/665719930880), sparse 0% (1546485760), duration 597, 20/20 MB/s
INFO: status: 3% (19983368192/665719930880), sparse 0% (1633570816), duration 910, 21/20 MB/s
INFO: status: 4% (26635468800/665719930880), sparse 0% (1761853440), duration 1226, 21/20 MB/s
INFO: status: 5% (33304215552/665719930880), sparse 0% (1856565248), duration 1539, 21/21 MB/s
INFO: status: 6% (39946682368/665719930880), sparse 0% (1926721536), duration 1851, 21/21 MB/s
INFO: status: 7% (46608023552/665719930880), sparse 0% (1981054976), duration 2146, 22/22 MB/s
INFO: status: 8% (53260320768/665719930880), sparse 0% (2067595264), duration 2457, 21/21 MB/s
INFO: status: 9% (59924873216/665719930880), sparse 0% (2144555008), duration 2766, 21/21 MB/s
INFO: status: 10% (66579791872/665719930880), sparse 0% (2231058432), duration 3084, 20/20 MB/s
INFO: status: 90% (599168057344/665719930880), sparse 3% (22405402624), duration 28702, 20/19 MB/s
INFO: status: 91% (605817405440/665719930880), sparse 3% (22931849216), duration 29030, 20/18 MB/s
INFO: status: 92% (612482154496/665719930880), sparse 3% (23351238656), duration 29371, 19/18 MB/s
INFO: status: 93% (619134779392/665719930880), sparse 3% (23770628096), duration 29696, 20/19 MB/s
INFO: status: 94% (625777901568/665719930880), sparse 3% (24198467584), duration 30019, 20/19 MB/s
INFO: status: 95% (632434458624/665719930880), sparse 3% (24617857024), duration 30348, 20/18 MB/s
INFO: status: 96% (639092326400/665719930880), sparse 3% (25054007296), duration 30678, 20/18 MB/s
INFO: status: 97% (645757140992/665719930880), sparse 3% (25465016320), duration 31007, 20/19 MB/s
INFO: status: 98% (652414222336/665719930880), sparse 3% (25901158400), duration 31330, 20/19 MB/s
INFO: status: 99% (659070976000/665719930880), sparse 3% (26333261824), duration 31661, 20/18 MB/s
INFO: status: 100% (665719930880/665719930880), sparse 4% (31834836992), duration 31980, 20/3 MB/s
INFO: transferred 665719 MB in 31980 seconds (20 MB/s)
INFO: archive file size: 346.57GB
INFO: delete old backup '/mnt/pve/nfsabs1-private/dump/vzdump-qemu-154-2013_06_08-07_38_53.vma.lzo'
INFO: Finished Backup of VM 154 (08:53:09)
INFO: Backup job finished successfully

 

[dietmar]

Also I detect that backups are more slow (biggest dump file size) with Proxmox v3.
And backups takes more disk space.

The versions you compare use exactly the same backup code.

 

[wahmed]

Are you sure about this ? In Version 3 when you do a backup in Snapshot mode and the VM is stopped the VM is put into a pre-start mode. That is the VM starts but does not boot. I think it also allocates the memory...

Ok, i believe you are right. Even though the VM is stopped, during backup it does go to pre launch mode which turns on the VM but not all the way. So....i tried something new. I allocated 10 GB of memory to the VM instead of 2GB then started a backup. Guess what...............drum roll.............................
THE BACKUP STARTED WITH 657mib/s READ AND 490 mib/s SPEED WITH WRITETHROUGH CACHE ENABLED!!! Although the speed dropped down and settled to 110 mib/s read, thats a quite a jump over NFS backup. And this worked for both FreeNAS and OmniOS NFS share over Gigabit network. Seems like i am going to be sticking with Writethrough cache for sometime.
Does it make sense to everybody?

 

[wahmed]

I know it is sort of old post, but just wanted to update of Backup speed dillemma.
I am glad to inform that my Proxmox Backup is now upto acceptable speed. It is steadily doing live backup around 80-85 MiB/s on shared storage. I have left FreeNAS behind(well not quite all the way) and setup OmniOS+Napp-It. I gave OmniOS a try couple months ago, but didnt quite settle down with that since i could not see acceptable speed. After weeks of tinkering this and that my proxmox VM backup seems to be headed to the right direction. Even if my current setup can keep the speed of 80-85 MiB/s steadiliy i am very much fine with that.

Thanks to "mir" for pointing me to OmniOS direction!