proxmox ve 3 slow backup to nfs share backed by iscsi

B

BloodyIron

Renowned Member
Jan 14, 2013
329
33
93
The storage for my VM disks right now is iSCSI backed so I can't do snapshots unfortunately. So I have set up a NFS share on another system, and the underlying storage for that is iSCSI backed also, but on completely separate drives on a completely separate iSCSI network.

The NFS share on the host is setup for async. This is also a very fresh installation of Proxmox ve 3.0, with the storage update.

When I try to do a backup of a KVM VM, whether it's running or not I get about 15MB/s / 15MB/s read/write.

My whole network is gigabit, and I'm certain I should be getting more than this, but I cannot yet determine why.

When I use the tool slurm on the NFS host I see that traffic bursts. When it bursts it can do upwards of 90-100MB/s, but then drops down to nearly nothing. So I suspect the 15/15 is simply an average over time. I do not know why it is bursting like this.

My googling has not yet turned anything up. Any ideas?
 
On the proxmox host, I changed directories to the place where the NFS backup share is mounted. I copied one of the partial backup files (about 400MB) in place to another file (test.dump) and it went for about 115MB/s solid in both directions according to slurm on the NFS host.

I suspect something with proxmox now.
 
Don't know the answer, just to point out that recent proxmox (>= 2.3) can do KVM snapshot backup on whatever storage/format, NFS included.
Also your VM could be very empty, so maybe it spends a lot of time reading empty sectors that are then not wrote to the backup, and there you get the slow transfer speed.
Also increasing MTU could help, you should find a way to test the speed it writes to nfs (i.e. save there a big file like an iso and calculate).
 
Unfortunately for me when I do iSCSI + LVM the snapshot is greyed out and I've seen this now on multiple nodes. Also, when I do a restore I get a much more substantial speed, like 80-110MB/s or so. It's peculiar the backup is so slow.

mmenaz said:
Don't know the answer, just to point out that recent proxmox (>= 2.3) can do KVM snapshot backup on whatever storage/format, NFS included.
Also your VM could be very empty, so maybe it spends a lot of time reading empty sectors that are then not wrote to the backup, and there you get the slow transfer speed.
Also increasing MTU could help, you should find a way to test the speed it writes to nfs (i.e. save there a big file like an iso and calculate).
Click to expand...
 
I've the feeling that you are mixing concept. Live Snapshot is one thing, backup in shapshot mode is another.
Live snapshot are only for qcow2 or some kind of storage (like Ceph RBD).
Backup in snapshot mode WAS for lvm, now is for whatever storage model you choose.
Go to Datacenter -> Backup -> Add, you will have a combo box with the available backup modes, none will ever be greyed out, if not supported will simply fail during backup
 
Well either way any method for backup runs at the above outlined ultra slow speed, that is for creating the backup, restoring is an appropriate speed.
 
You did not told me how big your HD are, and how filled. As I said, during backup ALL data is read and only used sectors are wrote, so if it takes 1 hour to read 1TB hd and writes to nfs only 1MB of data, then you have the mean transfer of 1MB/Hour.
You also did not told me how fast is simply copy of a big file from/to proxmox from/to nfs.
In addition, are you using compression and CPU is reaching it's limit? lzo is very fast in decompression (=restore), and slower in compression (=backup)
 
- The size of the VM disks are irrelevant, what is relevant is the performance I'm getting when performing the backup. But for informational sake, the Proxmox VE installation is very fresh, and the 3 VMs I'm working with are Ubuntu 13.04 Server that are also freshly installed, each 32GB allocated but only about 2GB of actual data on each

- When I restore I get 80MB/s-110MB/s so the NFS link is running at expected speeds

- I have tried the various compression options, none/LZO/GZIP and none have any impact on performance, additionally I have tried snapshot/suspend/offline and no difference there either, with the VM started or stopped. The Proxmox VE host is far from being close to hitting performance limits, the CPU in the host is an Intel Xeon E5-2620, and this isn't production at this time so the only task being performed at the time of the backup is the backup itself


As an additional note, I re-did one of the backups on one of the VMs to update the backup, and I saw 21MB/s-38MB/s, which if you'll note is substantially better than the 12MB/s-15MB/s I was seeing earlier. I have changed no settings so this increase in speed is confusing. This also was performed using the snapshot backup setting while the VM was running.


When I say snapshots I mean the actual snapshot feature, not backup while VM is running. A true snapshot is a partial backup, and takes up far less space. All of these backups are effectively duplicates of the disk image. That aside, I can live with this for now, but it's frustrating I cannot do true snapshots just because I'm using LVM.


mmenaz said:
You did not told me how big your HD are, and how filled. As I said, during backup ALL data is read and only used sectors are wrote, so if it takes 1 hour to read 1TB hd and writes to nfs only 1MB of data, then you have the mean transfer of 1MB/Hour.
You also did not told me how fast is simply copy of a big file from/to proxmox from/to nfs.
In addition, are you using compression and CPU is reaching it's limit? lzo is very fast in decompression (=restore), and slower in compression (=backup)
Click to expand...
 
Ok, so this seems to prove my point, try to fill one test vm with a lot of data (i.e. 30 gb out of 32 gb space) and see how it behaves.
How long does it take now to backup the vm? Could you post the related log file? You keep referring to bandwidth, and me insisting that is calculated only with a simple algorithm, data really moved / time taken, so if takes a lot of time to read 32 gb of hard disk and moves only 2GB of data, of sure it shows slow bandwidth.
Also, and I ask you again and for the last time, how does nfs behave copying to/from proxmox?
You are asking help and deciding what info are relevant to provide and what not, this is not a good attitude ;P.
 
Same speed issue with backup of RBD-backed VMs to NFS. It's regardless of data fill of machine.
INFO: starting new backup job: vzdump 103 --remove 0 --mode snapshot --compress lzo --storage backup --node vmc3-rndINFO: Starting Backup of VM 103 (qemu)
INFO: status = running
INFO: update VM 103: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/backup/dump/vzdump-qemu-103-2014_07_16-15_22_53.vma.lzo'
INFO: started backup task 'cf736580-0868-45e9-af49-305d5d6d5a5d'
INFO: status: 0% (166723584/34359738368), sparse 0% (135192576), duration 3, 55/10 MB/s
INFO: status: 1% (362545152/34359738368), sparse 0% (163307520), duration 11, 24/20 MB/s
INFO: status: 2% (703463424/34359738368), sparse 0% (199122944), duration 25, 24/21 MB/s
INFO: status: 3% (1036845056/34359738368), sparse 0% (215289856), duration 41, 20/19 MB/s
INFO: status: 4% (1381564416/34359738368), sparse 0% (239067136), duration 53, 28/26 MB/s
INFO: status: 5% (1739390976/34359738368), sparse 0% (257015808), duration 65, 29/28 MB/s
INFO: status: 6% (2067791872/34359738368), sparse 0% (257015808), duration 79, 23/23 MB/s
INFO: status: 7% (2436497408/34359738368), sparse 1% (392204288), duration 87, 46/29 MB/s
INFO: status: 8% (2760835072/34359738368), sparse 1% (392204288), duration 98, 29/29 MB/s
INFO: status: 9% (3119316992/34359738368), sparse 1% (392204288), duration 112, 25/25 MB/s
INFO: status: 10% (3451387904/34359738368), sparse 1% (400556032), duration 123, 30/29 MB/s
INFO: status: 11% (3793223680/34359738368), sparse 1% (437321728), duration 135, 28/25 MB/s
INFO: status: 12% (4141416448/34359738368), sparse 1% (515338240), duration 150, 23/18 MB/s
INFO: status: 13% (4490919936/34359738368), sparse 1% (650653696), duration 158, 43/26 MB/s
INFO: status: 14% (4828889088/34359738368), sparse 1% (650653696), duration 170, 28/28 MB/s
INFO: status: 15% (5175705600/34359738368), sparse 1% (650690560), duration 184, 24/24 MB/s
INFO: status: 16% (5517541376/34359738368), sparse 1% (667365376), duration 196, 28/27 MB/s
INFO: status: 17% (5848301568/34359738368), sparse 1% (667418624), duration 208, 27/27 MB/s
INFO: status: 18% (6189219840/34359738368), sparse 1% (667533312), duration 226, 18/18 MB/s
INFO: status: 19% (6598623232/34359738368), sparse 2% (810106880), duration 239, 31/20 MB/s
INFO: status: 20% (6882459648/34359738368), sparse 2% (818491392), duration 255, 17/17 MB/s
INFO: status: 21% (7222263808/34359738368), sparse 2% (818556928), duration 271, 21/21 MB/s
INFO: status: 22% (7561216000/34359738368), sparse 2% (818556928), duration 290, 17/17 MB/s
INFO: status: 23% (7906328576/34359738368), sparse 2% (827195392), duration 307, 20/19 MB/s
INFO: status: 24% (8271757312/34359738368), sparse 2% (827248640), duration 325, 20/20 MB/s
INFO: status: 25% (8595111936/34359738368), sparse 2% (827248640), duration 338, 24/24 MB/s
INFO: status: 26% (8988262400/34359738368), sparse 2% (981745664), duration 346, 49/29 MB/s
INFO: status: 27% (9293660160/34359738368), sparse 2% (988143616), duration 355, 33/33 MB/s
INFO: status: 28% (9628483584/34359738368), sparse 2% (989966336), duration 363, 41/41 MB/s
INFO: status: 29% (9979035648/34359738368), sparse 2% (999002112), duration 371, 43/42 MB/s
INFO: status: 30% (10359144448/34359738368), sparse 2% (1000693760), duration 380, 42/42 MB/s
INFO: status: 31% (10669654016/34359738368), sparse 2% (1003364352), duration 387, 44/43 MB/s
INFO: status: 32% (11028398080/34359738368), sparse 2% (1003622400), duration 395, 44/44 MB/s
INFO: status: 33% (11363745792/34359738368), sparse 2% (1004339200), duration 403, 41/41 MB/s
INFO: status: 34% (11705384960/34359738368), sparse 2% (1012723712), duration 419, 21/20 MB/s
INFO: status: 35% (12027887616/34359738368), sparse 2% (1012723712), duration 434, 21/21 MB/s
INFO: status: 36% (12393709568/34359738368), sparse 3% (1145958400), duration 444, 36/23 MB/s
INFO: status: 37% (12738101248/34359738368), sparse 3% (1149059072), duration 456, 28/28 MB/s
INFO: status: 38% (13070368768/34359738368), sparse 3% (1149104128), duration 468, 27/27 MB/s
INFO: status: 39% (13409648640/34359738368), sparse 3% (1149104128), duration 482, 24/24 MB/s
INFO: status: 40% (13762297856/34359738368), sparse 3% (1149104128), duration 494, 29/29 MB/s
INFO: status: 41% (14090633216/34359738368), sparse 3% (1149104128), duration 505, 29/29 MB/s
INFO: status: 42% (14480572416/34359738368), sparse 3% (1279340544), duration 516, 35/23 MB/s
INFO: status: 43% (14781054976/34359738368), sparse 3% (1288839168), duration 526, 30/29 MB/s
INFO: status: 44% (15146811392/34359738368), sparse 3% (1288839168), duration 538, 30/30 MB/s
INFO: status: 45% (15483797504/34359738368), sparse 3% (1288839168), duration 552, 24/24 MB/s
INFO: status: 46% (15817637888/34359738368), sparse 3% (1291788288), duration 563, 30/30 MB/s
INFO: status: 47% (16153509888/34359738368), sparse 3% (1291788288), duration 574, 30/30 MB/s
INFO: status: 48% (16622485504/34359738368), sparse 4% (1423216640), duration 587, 36/25 MB/s
INFO: status: 49% (16841375744/34359738368), sparse 4% (1429508096), duration 594, 31/30 MB/s
INFO: status: 50% (17201168384/34359738368), sparse 4% (1502236672), duration 606, 29/23 MB/s
INFO: status: 51% (17530290176/34359738368), sparse 4% (1502236672), duration 619, 25/25 MB/s
INFO: status: 52% (17880776704/34359738368), sparse 4% (1502236672), duration 631, 29/29 MB/s
INFO: status: 53% (18218614784/34359738368), sparse 4% (1502236672), duration 642, 30/30 MB/s
INFO: status: 54% (18555338752/34359738368), sparse 4% (1510612992), duration 655, 25/25 MB/s
INFO: status: 55% (18914410496/34359738368), sparse 4% (1644810240), duration 663, 44/28 MB/s
INFO: status: 56% (19260309504/34359738368), sparse 4% (1644810240), duration 675, 28/28 MB/s
INFO: status: 57% (19598409728/34359738368), sparse 4% (1644810240), duration 688, 26/26 MB/s
INFO: status: 58% (19942801408/34359738368), sparse 4% (1644810240), duration 701, 26/26 MB/s
INFO: status: 59% (20285227008/34359738368), sparse 4% (1644810240), duration 712, 31/31 MB/s
INFO: status: 60% (20622082048/34359738368), sparse 4% (1644810240), duration 725, 25/25 MB/s
INFO: status: 61% (20967653376/34359738368), sparse 5% (1775353856), duration 733, 43/26 MB/s
INFO: status: 62% (21321940992/34359738368), sparse 5% (1782124544), duration 745, 29/28 MB/s
INFO: status: 63% (21647458304/34359738368), sparse 5% (1782124544), duration 756, 29/29 MB/s
INFO: status: 64% (22014394368/34359738368), sparse 5% (1782124544), duration 770, 26/26 MB/s
INFO: status: 65% (22354198528/34359738368), sparse 5% (1782124544), duration 781, 30/30 MB/s
INFO: status: 66% (22679322624/34359738368), sparse 5% (1782124544), duration 792, 29/29 MB/s
INFO: status: 67% (23076667392/34359738368), sparse 5% (1911885824), duration 803, 36/24 MB/s
INFO: status: 68% (23378132992/34359738368), sparse 5% (1914474496), duration 813, 30/29 MB/s
INFO: status: 69% (23712694272/34359738368), sparse 5% (1914494976), duration 825, 27/27 MB/s
INFO: status: 70% (24077074432/34359738368), sparse 5% (1914494976), duration 837, 30/30 MB/s
INFO: status: 71% (24397938688/34359738368), sparse 5% (1914494976), duration 848, 29/29 MB/s
INFO: status: 72% (24753602560/34359738368), sparse 5% (1914494976), duration 862, 25/25 MB/s
INFO: status: 73% (25202589696/34359738368), sparse 5% (2048651264), duration 876, 32/22 MB/s
INFO: status: 74% (25437863936/34359738368), sparse 5% (2048651264), duration 886, 23/23 MB/s
INFO: status: 75% (25772228608/34359738368), sparse 5% (2048651264), duration 900, 23/23 MB/s
INFO: status: 76% (26137526272/34359738368), sparse 5% (2048651264), duration 917, 21/21 MB/s
INFO: status: 77% (26465075200/34359738368), sparse 5% (2048651264), duration 929, 27/27 MB/s
INFO: status: 78% (26825850880/34359738368), sparse 5% (2048651264), duration 942, 27/27 MB/s
INFO: status: 79% (27165130752/34359738368), sparse 5% (2048651264), duration 955, 26/26 MB/s
INFO: status: 80% (27491500032/34359738368), sparse 6% (2182832128), duration 963, 40/24 MB/s
INFO: status: 81% (27851423744/34359738368), sparse 6% (2182832128), duration 978, 23/23 MB/s
INFO: status: 82% (28179300352/34359738368), sparse 6% (2182832128), duration 990, 27/27 MB/s
INFO: status: 83% (28538568704/34359738368), sparse 6% (2182832128), duration 1003, 27/27 MB/s
INFO: status: 84% (28885975040/34359738368), sparse 6% (2182832128), duration 1017, 24/24 MB/s
INFO: status: 85% (29215686656/34359738368), sparse 6% (2182832128), duration 1029, 27/27 MB/s
INFO: status: 86% (29574037504/34359738368), sparse 6% (2317029376), duration 1037, 44/28 MB/s
INFO: status: 87% (29917511680/34359738368), sparse 6% (2317029376), duration 1051, 24/24 MB/s
INFO: status: 88% (30252269568/34359738368), sparse 6% (2317029376), duration 1063, 27/27 MB/s
INFO: status: 89% (30610096128/34359738368), sparse 6% (2317029376), duration 1077, 25/25 MB/s
INFO: status: 90% (30933254144/34359738368), sparse 6% (2317029376), duration 1089, 26/26 MB/s
INFO: status: 91% (31292325888/34359738368), sparse 6% (2317029376), duration 1101, 29/29 MB/s
INFO: status: 92% (31652642816/34359738368), sparse 7% (2454388736), duration 1109, 45/27 MB/s
INFO: status: 93% (31968854016/34359738368), sparse 7% (2454388736), duration 1121, 26/26 MB/s
INFO: status: 94% (32324386816/34359738368), sparse 7% (2454388736), duration 1134, 27/27 MB/s
INFO: status: 95% (32650166272/34359738368), sparse 7% (2462773248), duration 1145, 29/28 MB/s
INFO: status: 96% (33006616576/34359738368), sparse 7% (2462773248), duration 1157, 29/29 MB/s
INFO: status: 97% (33338228736/34359738368), sparse 7% (2462773248), duration 1170, 25/25 MB/s
INFO: status: 98% (33808449536/34359738368), sparse 7% (2605363200), duration 1182, 39/27 MB/s
INFO: status: 99% (34018754560/34359738368), sparse 7% (2605363200), duration 1192, 21/21 MB/s
INFO: status: 100% (34359738368/34359738368), sparse 7% (2621075456), duration 1210, 18/18 MB/s
INFO: transferred 34359 MB in 1210 seconds (28 MB/s)
INFO: archive file size: 5.21GB
INFO: Finished Backup of VM 103 (00:20:13)
INFO: Backup job finished successfully
TASK OK
Click to expand...
All NICs are 10Gbps.
 
Last edited:
Mitya said:
Same speed issue with backup of RBD-backed VMs to NFS. It's regardless of data fill of machine.

All NICs are 10Gbps.
Click to expand...

as your problem is different - see subject of the thread - you should open a new thread - and include more details about your storage configuration (VM and Backup target).
 
tom said:
as your problem is different - see subject of the thread - you should open a new thread - and include more details about your storage configuration (VM and Backup target).
Click to expand...
Actually, looks very similar: slow backup of network-storage-backed VM to network storage.
 
You must log in or register to reply here.
Top Bottom