Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/s

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

While it's a great platform in a number of ways and I'm impressed with it as a one-stop point for virtualisation, I'm not impressed with the ease at which changes are made (specifically, features removed), generally without explanation and not phased out. This is a prime example - add this new option (VMA's) fine - but why gut the old backup system - I'm sure it would still work fine as it's an OS feature, this was clearly just a dev decision not to allow it anymore. It makes backups less recoverable (can't extract the files manually anymore) - which as it turns out for me is a big deal.

The old backup system simple did not work for all storage types. The new system is superior and works on all storage types, but is currently optimized for fast backups.

 

[tom]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Instead of telling us that we are working on a ongoing beta crap software you should try to answer my basic question to go further here.

What RAID1 system do you test? Explain your hardware setup, explain your configuration and network and provide an example benchmark. Then we can compare it here with the results in our test-lab and see whats wrong.

As it works here it clear for me that you are doing something different.

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

The old backup system simple did not work for all storage types. The new system is superior and works on all storage types, but is currently optimized for fast backups.

Understand the need to backup all types of storage systems with snapshots. Offering that is great. Removing the old option that was just as fast at backing up (within a few seconds on my servers) and replacing it with a new custom system that has a slower/unoptimised restore isn't better, it's a step back.

Instead of telling us that we are working on a ongoing beta crap software you should try to answer my basic question to go further here.

What RAID1 system do you test? Explain your hardware setup, explain your configuration and network and provide an example benchmark. Then we can compare it here with the results in our test-lab and see whats wrong.

As it works here it clear for me that you are doing something different.

Clearly I'm doing something different to you. But still, its pretty straight forward. I've seen a test here (which helped) that showed a restore going to a RAID6 array. I've asked several times about any performance testing to RAID1 or even a single dedicated disk and have been told nothing.

I've tried a few raid cards (an intel SRCSATAWB/LSI card in one test server / a RocketRAID 3520 in another), and also directly attatched drives. With write through cache (no cache battery) the performance is very poor (25mb/s). Restoring to a directly attatched disk varies at 12-15MB/s. With write back cache on (should use cache battery) performance is far better at 50mb/s but still short of usual full-speed file write (100mb/s).

I've posted a log file already and explained all I can about the testing I've done. The only hope I can see at getting reasonable restore speeds (100MB/s+) is to restore to SSD first with WB cache, or a RAID5 or RAID0 array.

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

and replacing it with a new custom system that has a slower/unoptimised restore isn't better, it's a step back.

So far you are the only one claiming the restore is slower. I also can't reproduce it here. Somebody else notice slow restore speed?

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

So far you are the only one claiming the restore is slower. I also can't reproduce it here. Somebody else notice slow restore speed?

Really, how often do you restore, and how long has this system been out? Here's my point, the LVM backup option should have never been removed until adequate real world testing of the new backup method had been done with solid community acceptance. My other concern with the poor write performance was fragmentation on the filesystem, as it has become pretty clear to me that the restore is occuring in a non-contiguous manner.

Check the number of extents (fragments) on some of your restored files (eg filefrag vm-101-disk-2.raw).

Original File;
vm-101-disk-2.raw: 34 extents found

Restored File;
vm-101-disk-2.raw: 10042 extents found

The restored file should be in a single fragment, or perhaps just a few. The restored file has an excess 10,000 fragments added by the restore process, and this is over only 1.1g of real data (a 4g sparse file).

Seems to me like a filesystem fragment/seek every 100KB is going to cause some performance issues on the restored VM.

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

The restored file should be in a single fragment, or perhaps just a few. The restored file has an excess 10,000 fragments added by the restore process, and this is over only 1.1g of real data (a 4g sparse file).

Seems to me like a filesystem fragment/seek every 100KB is going to cause some performance issues on the restored VM.

Sorry, I simply can't reproduce those number here, and restore I always fast here. Also, the number of fragment depend on the utilization of your file system, and the block size used by the copy/restore. You probably never get a single fragment (a simple 'cp' of a raw file produces 1000 fragments on my local system, and 'qemu-image convert' produces even more fragment due to smaller clock size. Besides, performance is still good.)

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Sorry, I simply can't reproduce those number here, and restore I always fast here.

What sort of local storage are you using? I would not be surprised that you'd not see a problem with low compression and a RAID5 array. As I've asked many times now - try slower RAID1 or a local direct disk. My images are commonly compressed at 300-500% and so local write speed is critical.

I'd too would love to hear anyone with more testing/feedback on restore performance one way or the other (actual local write speeds, not 'uncompressed' speeds) as all my testing on various systems are coming up badly.

Copying to a xfs filesystem I get one extent. If I copy the file out from the destination to an ext4 partition I get;

vm-101-disk-2.raw: 32 extents found

Which is around the same amount of extents that were on the original file. I did get a lot more extents on an ext3 filesystem when doing a cp, however they were 'clean' and the file stored quickly (look at the sector allocation with -v). With the restored file it isn't clean, there are sector allocations all over the shop in a very uneven manner.

If you can't be bothered trying a slower write array (eg RAID1), It would be quite easy to prove I'm talking crap here on any filesystem would be by posting a few benchmarks of restore write performance in proxmox vs extracing a file to it from a highly compressed source (to supply enough data to write fast to the destination), but I don't see any of that testing.

I'm still not saying restore performance was great before with LVM, as for whatever reason it wasn't - but now we don't have an option to directly extract the files from the .tar.gz - we must either use the proxmox restore system or utilize another backup system completely for fast restores.

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

What sort of local storage are you using? I would not be surprised that you'd not see a problem with low compression and a RAID5 array. As I've asked many times now - try slower RAID1 or a local direct disk.

We test on many different system (fast RAID, slow RAID, no RAID, SSD, ...)

Anyway, you need to find a way to reproduce the behavior and document that exactly. Everything I test is fast.

 

[mir]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

I have never heard of that writing to RAID5 should be faster than writing to RAID1. Do you have any proof to this statement?

 

[Trimmings]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

We test on many different system (fast RAID, slow RAID, no RAID, SSD, ...)

Anyway, you need to find a way to reproduce the behavior and document that exactly. Everything I test is fast.

I think we will have to wait for more feedback - I can't test any more than I already have and it sounds like you've done a lot of testing. Still, no write performance figures from your testing other than 'fast' for no raid and 'slow' raid (assuming you mean RAID1 by slow raid, everything else is faster).

Regardless, I'd still like to put my hand up for the LVM backup option to be allowed again (as an option at least). I think this is important at the very minumum to make the backups more accessible (as in non-proprietary). I'd probably even fucking pay for it to be put back in as an option. Otherwise, I guess I'm going to have to do LVM backups manually.

I have never heard of that writing to RAID5 should be faster than writing to RAID1. Do you have any proof to this statement?

Uh yeah. Its quite well mathematically documented. Assuming you are using a disk set of greater than 2 (Raid 5 with 2 disks would be funny lol). http://en.wikipedia.org/wiki/RAID .

 

[dietmar]

Re: Super slow vma backup restore - about 1.5MB/s with drives that will copy at 90MB/

Uh yeah. Its quite well mathematically documented.

That mathematical model is a bit simplified, an only models max. write speed.