Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/s

[Trimmings]

As the subject. Reading from disk at 3-700KB/s, destination maxed out at 1.5MB/s. When doing a straight copy it will do 90MB/s happily with same source and destination drives.

This is really, really broken. Not sure what the point of a backup is if it will take three days to restore.

- - - Updated - - -

Destination drive;


Restore of VMA backup;
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 59.20 0.00 104.20 0.00 1475.20 28.31 0.99 9.39 9.44 98.38


Direct copy;
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdb 0.00 0.00 0.60 140.20 2.40 71782.40 1019.67 143.50 1015.75 7.10 100.00

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

What do you talk about exactly?

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

What do you talk about exactly?

I'm restoring a backup. On the same hardware restore used to be slow with the old LVM snapshot backups but much faster than this (15-20MB/s). It's not ten times slower at least. I'm showing disk stats (iostat -d -k -x 5) because the destination drive is getting hammered,the cpu is only doing about 1%.

- - - Updated - - -

What do you talk about exactly?

I'm restoring a backup. On the same hardware restore used to be slow with the old LVM snapshot backups but much faster than this (15-20MB/s). It's not ten times slower at least. I'm showing disk stats (iostat -d -k -x 5) because the destination drive is getting hammered,the cpu is only doing about 1%.

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

restore vma archive: zcat /mnt/ntfs/dump/vzdump-qemu-102-2013_05_09-22_21_32.vma.gz|vma extract -v -r /var/tmp/vzdumptmp3288.fifo - /var/tmp/vzdumptmp3288
CFG: size: 557 name: qemu-server.conf
DEV: dev_id=1 size: 107390304256 devname: drive-virtio1
DEV: dev_id=2 size: 85914098688 devname: drive-virtio2
DEV: dev_id=3 size: 4294967296 devname: drive-virtio3
DEV: dev_id=4 size: 10737418240 devname: drive-virtio4
CTIME: Thu May 9 22:21:32 2013
Formatting '/mnt/disk2/images/102/vm-102-disk-1.vmdk', fmt=vmdk size=107390304256 compat6=off
new volume ID is 'disk2:102/vm-102-disk-1.vmdk'
map 'drive-virtio1' to '/mnt/disk2/images/102/vm-102-disk-1.vmdk' (write zeros = 0)
Formatting '/mnt/disk2/images/102/vm-102-disk-2.vmdk', fmt=vmdk size=85914098688 compat6=off
new volume ID is 'disk2:102/vm-102-disk-2.vmdk'
map 'drive-virtio2' to '/mnt/disk2/images/102/vm-102-disk-2.vmdk' (write zeros = 0)
Formatting '/mnt/disk2/images/102/vm-102-disk-3.raw', fmt=raw size=4294967296
new volume ID is 'disk2:102/vm-102-disk-3.raw'
map 'drive-virtio3' to '/mnt/disk2/images/102/vm-102-disk-3.raw' (write zeros = 0)
Formatting '/mnt/disk2/images/102/vm-102-disk-4.vmdk', fmt=vmdk size=10737418240 compat6=off
new volume ID is 'disk2:102/vm-102-disk-4.vmdk'
map 'drive-virtio4' to '/mnt/disk2/images/102/vm-102-disk-4.vmdk' (write zeros = 0)
progress 1% (read 2083389440 bytes, duration 2193 sec)
progress 2% (read 4166778880 bytes, duration 4255 sec)
progress 3% (read 6250168320 bytes, duration 6651 sec)
progress 4% (read 8333492224 bytes, duration 8865 sec)
progress 5% (read 10416881664 bytes, duration 11084 sec)
progress 6% (read 12500271104 bytes, duration 13312 sec)
progress 7% (read 14583595008 bytes, duration 15536 sec)
progress 8% (read 16666984448 bytes, duration 17731 sec)
progress 9% (read 18750373888 bytes, duration 19952 sec)
progress 10% (read 20833697792 bytes, duration 22140 sec)
progress 11% (read 22917087232 bytes, duration 24182 sec)
progress 12% (read 25000476672 bytes, duration 26220 sec)
progress 13% (read 27083800576 bytes, duration 27253 sec)
progress 14% (read 29167190016 bytes, duration 27755 sec)
progress 15% (read 31250579456 bytes, duration 29995 sec)
progress 16% (read 33333903360 bytes, duration 32209 sec)
progress 17% (read 35417292800 bytes, duration 34439 sec)
progress 18% (read 37500682240 bytes, duration 35922 sec)
progress 19% (read 39584006144 bytes, duration 36078 sec)
progress 20% (read 41667395584 bytes, duration 36078 sec)
progress 21% (read 43750785024 bytes, duration 36078 sec)
progress 22% (read 45834108928 bytes, duration 36078 sec)
progress 23% (read 47917498368 bytes, duration 36078 sec)
progress 24% (read 50000887808 bytes, duration 36079 sec)
progress 25% (read 52084211712 bytes, duration 36079 sec)
progress 26% (read 54167601152 bytes, duration 36079 sec)
progress 27% (read 56250990592 bytes, duration 36079 sec)
progress 28% (read 58334314496 bytes, duration 36079 sec)
progress 29% (read 60417703936 bytes, duration 36079 sec)
progress 30% (read 62501093376 bytes, duration 36079 sec)
progress 31% (read 64584417280 bytes, duration 36080 sec)
progress 32% (read 66667806720 bytes, duration 36080 sec)
progress 33% (read 68751196160 bytes, duration 36080 sec)
progress 34% (read 70834520064 bytes, duration 36080 sec)
progress 35% (read 72917909504 bytes, duration 36080 sec)
progress 36% (read 75001298944 bytes, duration 36080 sec)
progress 37% (read 77084622848 bytes, duration 36080 sec)
progress 38% (read 79168012288 bytes, duration 36080 sec)
progress 39% (read 81251401728 bytes, duration 36081 sec)
progress 40% (read 83334725632 bytes, duration 36081 sec)
progress 41% (read 85418115072 bytes, duration 36081 sec)
progress 42% (read 87501504512 bytes, duration 36081 sec)
progress 43% (read 89584828416 bytes, duration 36081 sec)
progress 44% (read 91668217856 bytes, duration 36081 sec)
progress 45% (read 93751607296 bytes, duration 36081 sec)
progress 46% (read 95834931200 bytes, duration 36082 sec)
progress 47% (read 97918320640 bytes, duration 36082 sec)
progress 48% (read 100001710080 bytes, duration 36082 sec)
progress 49% (read 102085033984 bytes, duration 36082 sec)
progress 50% (read 104168423424 bytes, duration 36082 sec)
progress 51% (read 106251812864 bytes, duration 36082 sec)
progress 52% (read 108335136768 bytes, duration 37072 sec)
progress 53% (read 110418526208 bytes, duration 39334 sec)
progress 54% (read 112501915648 bytes, duration 41200 sec)
progress 55% (read 114585239552 bytes, duration 42042 sec)
progress 56% (read 116668628992 bytes, duration 42241 sec)
progress 57% (read 118752018432 bytes, duration 42241 sec)
progress 58% (read 120835342336 bytes, duration 42241 sec)
progress 59% (read 122918731776 bytes, duration 42357 sec)
progress 60% (read 125002121216 bytes, duration 42357 sec)
progress 61% (read 127085445120 bytes, duration 42358 sec)
progress 62% (read 129168834560 bytes, duration 42358 sec)
progress 63% (read 131252224000 bytes, duration 42358 sec)
progress 64% (read 133335547904 bytes, duration 42358 sec)
progress 65% (read 135418937344 bytes, duration 42358 sec)
progress 66% (read 137502326784 bytes, duration 42358 sec)
progress 67% (read 139585650688 bytes, duration 42358 sec)
progress 68% (read 141669040128 bytes, duration 42359 sec)
progress 69% (read 143752429568 bytes, duration 42457 sec)
progress 70% (read 145835753472 bytes, duration 42457 sec)
progress 71% (read 147919142912 bytes, duration 42458 sec)
progress 72% (read 150002532352 bytes, duration 42458 sec)
progress 73% (read 152085856256 bytes, duration 42458 sec)
progress 74% (read 154169245696 bytes, duration 42458 sec)
progress 75% (read 156252635136 bytes, duration 42458 sec)
progress 76% (read 158335959040 bytes, duration 42458 sec)
progress 77% (read 160419348480 bytes, duration 42458 sec)
progress 78% (read 162502737920 bytes, duration 42459 sec)
progress 79% (read 164586061824 bytes, duration 42459 sec)
progress 80% (read 166669451264 bytes, duration 42459 sec)
progress 81% (read 168752840704 bytes, duration 42459 sec)
progress 82% (read 170836164608 bytes, duration 42459 sec)
progress 83% (read 172919554048 bytes, duration 42459 sec)
progress 84% (read 175002943488 bytes, duration 42459 sec)
progress 85% (read 177086267392 bytes, duration 42460 sec)
progress 86% (read 179169656832 bytes, duration 42460 sec)
progress 87% (read 181253046272 bytes, duration 42460 sec)
progress 88% (read 183336370176 bytes, duration 42460 sec)
progress 89% (read 185419759616 bytes, duration 42460 sec)
progress 90% (read 187503149056 bytes, duration 42460 sec)
progress 91% (read 189586472960 bytes, duration 42460 sec)
progress 92% (read 191669862400 bytes, duration 42461 sec)
progress 93% (read 193753251840 bytes, duration 42461 sec)
progress 94% (read 195836575744 bytes, duration 42464 sec)
progress 95% (read 197919965184 bytes, duration 42866 sec)
progress 96% (read 200003354624 bytes, duration 45122 sec)
progress 97% (read 202086678528 bytes, duration 46013 sec)
progress 98% (read 204170067968 bytes, duration 48185 sec)
progress 99% (read 206253457408 bytes, duration 50361 sec)
progress 100% (read 208336781312 bytes, duration 50361 sec)
total bytes read 208336846848, sparse bytes 160066206720 (76.8%)
space reduction due to 4K zero bocks 1.11%
TASK OK

- - - Updated - - -

p.s. getting this while browsing the forum (when logged in especially on this post)

HTTP Error 500 (Internal Server Error): An unexpected condition was encountered while the server was attempting to fulfil the request.

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

This backup is created when the VM was actively writing data?

For testing, try to make a backup when the VM is stopped, the restore that - is that faster?

Is it faster when you restore to local storage?

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

This backup is created when the VM was actively writing data?

For testing, try to make a backup when the VM is stopped, the restore that - is that faster?

Is it faster when you restore to local storage?

There is nothing running on the drives. Since clearly there is a problem with the setup and I imagine in development devs at least once ran a restore and it went at some sort of acceptable speed I tested on two new servers and found the restore was faster. In the end, it turns out that using ext4 destroys restore performance with the new vma backup method.

Destination drive;
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util


ext4;
sdb 0.00 69.20 0.00 123.80 0.00 1372.80 22.18 1.00 8.07 7.99 98.88


ext3;
sdb 0.00 4892.40 0.00 1557.00 0.00 25785.60 33.12 1.11 0.71 0.53 82.36


Disk ability (direct copy);
sdb 0.00 0.00 0.60 140.20 2.40 71782.40 1019.67 143.50 1015.75 7.10 100.00

So, ext4 is basically fucked for restores on the new format which is great given I've updated several production servers to v2.3 that are all operating on ext4. Restores are just too slow.

Ext3 is better, but still a third of the performance the drive's ability. What on earth is going on here - I'd rather have slower backups than slower restores, restores need to be fast if you are to recover a server, and you can't manually extract the files from the gz yourself anymore with vma format...

1) Is it possible to use the old backup process in V 2.3+ (LVM)
2) Can the restore performance be increased (the restore seems to be trying to be 'nice' to the destination disk with regards to wait times)
3) Can we please fix performance for vma restores on ext4

 

[martin]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

..

1) Is it possible to use the old backup process in V 2.3+ (LVM)

no. test with latest kernel (3.0)

2) Can the restore performance be increased (the restore seems to be trying to be 'nice' to the destination disk with regards to wait times)

its fast here. so there is something different on your side. give details about your hardware, raid controller, etc.

results of 'pveperf'?

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

oh I just did a test with ext2;

Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdc 0.00 2.00 0.00 413.00 0.00 5702.40 27.61 0.91 2.21 2.06 85.18

Yeah about 5.8Mb/s for ext2 as a destination.

 

[tom]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

if you don´t provide the requested info I cannot give further hints.

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

There is no raid controller in use except for the installation, this is an i5, 16GB test box and I'm writing direct to sata connected hard disks to eliminate any raid performance impacts.

CPU BOGOMIPS: 26340.44
REGEX/SECOND: 1453746
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 125.61 MB/sec
AVERAGE SEEK TIME: 10.08 ms
FSYNCS/SECOND: 782.33
DNS EXT: 200.98 ms
DNS INT: 20.46 ms (localhost)



I'm not going to upgrade this box yet as the V3 is still buggy (I tested it already, the browser support for chrome isn't working).

You say it's 'fast' to restore, I've tried several other machines now and they are all the same (2.3 and 3) - the restore speed even to ext3 is about 1/3 of the write speed thoroughput capabilities of the destination.

Can you confirm in your testing you get somewhere close to 100% of the destination performance? That is a drive that does 100mb/s with a direct file write restores backups at 100mb/s? As I said, at least with the previous format I could manually extract the files from the gz and achieve full restore speed, that option is now gone.

 

[tom]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

There is no raid controller in use except for the installation, this is an i5, 16GB test box and I'm writing direct to sata connected hard disks to eliminate any raid performance impacts.

CPU BOGOMIPS: 26340.44
REGEX/SECOND: 1453746
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 125.61 MB/sec
AVERAGE SEEK TIME: 10.08 ms
FSYNCS/SECOND: 782.33
DNS EXT: 200.98 ms
DNS INT: 20.46 ms (localhost)



I'm not going to upgrade this box yet as the V3 is still buggy (I tested it already, the browser support for chrome isn't working).

Chrome should work without any problems. I do almost all my work with Chrome.

You say it's 'fast' to restore, I've tried several other machines now and they are all the same (2.3 and 3) - the restore speed even to ext3 is about 1/3 of the write speed thoroughput capabilities of the destination.

Can you confirm in your testing you get somewhere close to 100% of the destination performance? That is a drive that does 100mb/s with a direct file write restores backups at 100mb/s? As I said, at least with the previous format I could manually extract the files from the gz and achieve full restore speed, that option is now gone.

Here is a restore log from. the file is located on a NFS server, connected with Gbit speed.

Destination is local storage, formatted with ext4. 5 years old disk, connected to a Adaptec 6805, raid6.

restore vma archive: lzop -d -c /mnt/pve/nfs-tepm/dump/vzdump-qemu-210-2013_05_19-18_45_59.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp3220.fifo - /var/tmp/vzdumptmp3220
CFG: size: 755 name: qemu-server.conf
DEV: dev_id=1 size: 53687091200 devname: drive-virtio0
DEV: dev_id=2 size: 107374182400 devname: drive-virtio1
DEV: dev_id=3 size: 4294967296 devname: drive-virtio2
CTIME: Sun May 19 18:46:01 2013
Formatting '/var/lib/vz/images/102/vm-102-disk-2.raw', fmt=raw size=53687091200
new volume ID is 'local:102/vm-102-disk-2.raw'
map 'drive-virtio0' to '/var/lib/vz/images/102/vm-102-disk-2.raw' (write zeros = 0)
Formatting '/var/lib/vz/images/102/vm-102-disk-3.raw', fmt=raw size=107374182400
new volume ID is 'local:102/vm-102-disk-3.raw'
map 'drive-virtio1' to '/var/lib/vz/images/102/vm-102-disk-3.raw' (write zeros = 0)
Formatting '/var/lib/vz/images/102/vm-102-disk-4.raw', fmt=raw size=4294967296
new volume ID is 'local:102/vm-102-disk-4.raw'
map 'drive-virtio2' to '/var/lib/vz/images/102/vm-102-disk-4.raw' (write zeros = 0)
progress 1% (read 1653604352 bytes, duration 13 sec)
progress 2% (read 3307143168 bytes, duration 26 sec)
progress 3% (read 4960747520 bytes, duration 38 sec)
progress 4% (read 6614286336 bytes, duration 51 sec)
progress 5% (read 8267825152 bytes, duration 65 sec)
...
progress 87% (read 143859974144 bytes, duration 1056 sec)
progress 88% (read 145513512960 bytes, duration 1069 sec)
progress 89% (read 147167117312 bytes, duration 1082 sec)
progress 90% (read 148820656128 bytes, duration 1097 sec)
progress 91% (read 150474194944 bytes, duration 1114 sec)
progress 92% (read 152127799296 bytes, duration 1128 sec)
progress 93% (read 153781338112 bytes, duration 1142 sec)
progress 94% (read 155434876928 bytes, duration 1158 sec)
progress 95% (read 157088481280 bytes, duration 1173 sec)
progress 96% (read 158742020096 bytes, duration 1185 sec)
progress 97% (read 160395558912 bytes, duration 1200 sec)
progress 98% (read 162049163264 bytes, duration 1212 sec)
progress 99% (read 163702702080 bytes, duration 1222 sec)
progress 100% (read 165356240896 bytes, duration 1238 sec)
total bytes read 165356240896, sparse bytes 18709979136 (11.3%)
space reduction due to 4K zero bocks 1.08%
TASK OK

or in other words: 165 GB in 20 minutes.

 

[tom]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

...

or in other words: 165 GB in 20 minutes.

forgot to mention:

ls -alh vzdump-qemu-210-2013_05_19-18_45_59.vma.lzo 

-rw-r--r-- 1 root root 90G May 19 19:07 vzdump-qemu-210-2013_05_19-18_45_59.vma.lzo

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Chrome should work without any problems. I do almost all my work with Chrome.

Here is a restore log from. the file is located on a NFS server, connected with Gbit speed.

Destination is local storage, formatted with ext4. 5 years old disk, connected to a Adaptec 6805, raid6.

or in other words: 165 GB in 20 minutes.

Re chrome - I dunno, I get this: "Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data."
IE is OK and so is firefox (and I'm not the only one http://forum.proxmox.com/threads/13732-Proxmox-VE-3-0-RC1-released!?p=73964#post73964). I've tried clearing my cache in chrome but no fix.

Re your restore speed - I can only imagine what you'd get uncompressing a stream to that from a compressed source but I imagine you might get around 300-390Mb/s (assuming 5x drives, you'd get 3x individual disk speed for read/write performance and 2x parity)? Your speed seems to be about 130MB/s which could be a third of the speed of the array.

Perhaps you could run a restore and take a look at iostat to see wether you think it is limited by source or destination (iostat -d -k -x 5) - my source is being hardly utilised (10-20%).

Of course you may not see a problem with this setup but raid1 systems will not restore well if restricted to 1/3 of available write speeds on magnetic disks.

 

[tom]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Re chrome - I dunno, I get this: "Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data."
IE is OK and so is firefox (and I'm not the only one http://forum.proxmox.com/threads/13732-Proxmox-VE-3-0-RC1-released!?p=73964#post73964). I've tried clearing my cache in chrome but no fix.

Re your restore speed - I can only imagine what you'd get uncompressing a stream to that from a compressed source but I imagine you might get around 300-390Mb/s (assuming 5x drives, you'd get 3x individual disk speed for read/write performance and 2x parity)? Your speed seems to be about 130MB/s which could be a third of the speed of the array.

your calculation does not include the fact that my backup is on NAS, connected with 1 Gbit. So in my case, the network speed is the first limiting factor. You report a speed of 1,5 MB/s - this is something I never got here there must be something really wrong on your side. dig deeper.

Perhaps you could run a restore and take a look at iostat to see wether you think it is limited by source or destination (iostat -d -k -x 5) - my source is being hardly utilised (10-20%).

Of course you may not see a problem with this setup but raid1 systems will not restore well if restricted to 1/3 of available write speeds on magnetic disks.

raid1 speed is also OK here. please note, backup/restore process should never use all your IO performance, its always a low priority task in order to NOT affect the rest of the services running on your Proxmox VE system.

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

your calculation does not include the fact that my backup is on NAS, connected with 1 Gbit. So in my case, the network speed is the first limiting factor. You report a speed of 1,5 MB/s - this is something I never got here there must be something really wrong on your side. dig deeper.

raid1 speed is also OK here. please note, backup/restore process should never use all your IO performance, its always a low priority task in order to NOT affect the rest of the services running on your Proxmox VE system.

Yeah very true, forgot about your network but in my case backups are generally compressed 2-300% and so the source can still be only 100MB/s and restore could in theory occur at 2-300MB/s. Have you done a restore to RAID1 mechanical/magnetic disks going at 100MB/s+? I can't get restores happening at greater than 30MB/s at the moment with a single locally attached disk. The destination seems to be the limitation as CPU use is low, source disk is also uncongested.

My greatest concern is getting all virtual servers restored in case of server/system failure as quickly as possible. When we're talking about a mix of servers that may be up to 1TB size, throughput is important. It doesn't matter for us if we cause poor performance on the host, we will simply need to restore all servers as quickly as possible. In this case, if this is perhaps a priority thing, is there not an option to allow the restore to be 'high priority' and to allow it to fully saturate the IO of the destination disk?

Thanks

 

[tom]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Yeah very true, forgot about your network but in my case backups are generally compressed 2-300% and so the source can still be only 100MB/s and restore could in theory occur at 2-300MB/s. Have you done a restore to RAID1 mechanical/magnetic disks going at 100MB/s+? I can't get restores happening at greater than 30MB/s at the moment with a single locally attached disk. The destination seems to be the limitation as CPU use is low, source disk is also uncontested.

My greatest concern is getting all virtual servers restored in case of server/system failure as quickly as possible. When we're talking about a mix of servers that may be up to 1TB size, throughput is important. It doesn't matter for us if we cause poor performance on the host, we will simply need to restore all servers as quickly as possible. In this case, if this is perhaps a priority thing, is there not an option to allow the restore to be 'high priority' and to allow it to fully saturate the IO of the destination disk?

Thanks

You talk about hardware raid1 (what controller to you use, provide your 'pveperf') or md-raid1? In the case of md-raid, read http://pve.proxmox.com/wiki/Software_RAID

- - - Updated - - -

just to note, never use gz if you want speed - lzo is MUCH faster.

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

You talk about hardware raid1 (what controller to you use, provide your 'pveperf') or md-raid1? In the case of md-raid, read http://pve.proxmox.com/wiki/Software_RAID

- - - Updated - - -

just to note, never use gz if you want speed - lzo is MUCH faster.

Tom, wether using gz or lzo this is entirely cpu related. I understand lzo uses less cpu - there is no cpu limitation here, this is a disk write performance limitation. Using gz in fact should assure a faster restore time as the performance limitation should not be the destination media, it should be the source (backup) media. I can understand all this talk about fast backups. Fast backups are nice, fast restores are more important.

I'm talking about testing on a direct restore on a new basic system with a directly attached disk, no raid. I don't understand why the performance is so poor when doing this and I still don't have any answer short of it being a 'low priority task'. All testing I've done so far seems to indicate that restores are done with a fraction of the destination/recovery drive's performance (the source backup media and cpu are not saturated). A way to allow the recovery to become 'high priority' should be devised if this is in fact what is slowing restores by 2-300%

I can assure you a critical server recovery isn't a low priority task and in my case at least the destination storage isn't shared with other hosts in a way that will impact their performance should the IO become saturated. Since raid-1 isn't in any significant write performance way different to a directly attatched single disk then surely this test should be worth something.

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

So, I've done some testing on RAID1 backed ssd on a production system. It seems to indicate that without a writeback cache the restore performance is significantly slower (50%).

RAID1 SSD
Write Back
sdb 0.00 13343.20 0.00 4313.40 0.00 70626.40 32.75 0.39 0.09 0.07 29.68
Write Through
sdb 0.00 6278.60 0.00 2040.80 0.00 33289.60 32.62 0.80 0.39 0.29 58.88

Source;
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sdd 0.00 0.00 88.20 0.00 11289.60 0.00 256.00 0.11 1.26 1.00 8.86


Relevant top processes during restore;
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
847771 root 20 0 4228 608 420 R 22 0.0 0:42.06 gzip
847772 root 20 0 88232 8020 3456 S 19 0.1 0:41.90 vma


Can you guys please do some more work on recovery of backups. At least allow some options to change priority of the restoration task rather than throwing the problem to the raid card to sort out.

I'm going to have to consider setting up snapshots myself in the meantime, which is a huge pain in the ass especially considering the previous backup system allowed me to restore backups from a snapshot directly (by uncompressing the gz file, giving 'real' disk speeds).

Again this may not be an issue for you guys (using raid cards that have battery backup so you can switch on buffered writes) but I need to be able to provide recovery windows to customers and this change has just royally screwed all windows previously defined by testing on the previous backup platform

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Again this may not be an issue for you guys (using raid cards that have battery backup so you can switch on buffered writes) but I need to be able to provide recovery windows to customers and this change has just royally screwed all windows previously defined by testing on the previous backup platform

You obviously provide commercial services using Proxmox VE. If you want commercial support, you should consider to buy a support subscription.
Another option would be that you try to optimize the code yourself and send patches for inclusion - see http://pve.proxmox.com/wiki/Developer_Documentation

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

You obviously provide commercial services using Proxmox VE. If you want commercial support, you should consider to buy a support subscription.
Another option would be that you try to optimize the code yourself and send patches for inclusion - see http://pve.proxmox.com/wiki/Developer_Documentation

Yeah at this stage proxmox is a system I'm considering for broader deployment that it currently is being used for. But, anyway if you're interested - I'll give you my opinion. I don't 'sell' VM's, but I do use virtualisation for onsite servers at customer premises, currently with proxmox only at a few sites. I'd certainly consider supporting proxmox, I already have considered it. I'm a software engineer and so I appreciate that the system is open source and I could do whatever I want myself with the code, but that's not the point. This is an 'off the shelf' product from my perspective.

While it's a great platform in a number of ways and I'm impressed with it as a one-stop point for virtualisation, I'm not impressed with the ease at which changes are made (specifically, features removed), generally without explanation and not phased out. This is a prime example - add this new option (VMA's) fine - but why gut the old backup system - I'm sure it would still work fine as it's an OS feature, this was clearly just a dev decision not to allow it anymore. It makes backups less recoverable (can't extract the files manually anymore) - which as it turns out for me is a big deal. The forum advice I've had in the past to 'always update to the latest software' really isn't in line with a commercial offering, where you have longer release cycles with heavy testing and QA done on those releases.

My perspective is that if the developers desire a commercial support relationship, I'd expect that the product's feature set should be set like a commercial product and not like an ongoing beta.

So believe it or not, I'm hoping that my feedback/testing (at considerable time investment myself now) will lead to improvements that will improve the product to the point that it is more like a commercial product. Thanks for at the very least having people active on the forums, and thanks for any efforts put in especially given possible language barriers.