Poor hard drive performance...

[oeginc]

We had a server crash recently (power supply died, turns out we were running 150 watt power supplies in our 3U rackmount servers and they just couldn't keep up with 6 hard drives, raid card, etc... Surprised it worked at all! LOL!)....

Anyways, we've been having some big problems keeping up with disk I/O, so I thought while the server was down I'd play around a little bit.

I don't have the exact PVE version information, but the server (as it stood) was originally a v1.5 install that had been apt-get upgrade'd to v1.6 and was running a 2.6.18-4 kernel.

The hardware:
Gigabyte motherboard
AMD Phenom x4 9850 quad-core processor
8GB RAM
Highpoint 2640x4 RAID Controller card (I know, I have adaptec 5805z's here just waiting to be swapped out).
1 Hitachi 500GB SATA2 7,200 RPM Drive - Boot & Proxmox
1 Western Digital Black 2TB SATA2 7,200 RPM Drive - VM Backups
4 Segate 500GB SATA2 7,2000 RPM Drives in RAID 10 - Virtual Machines

In the host running dbench on the RAID10 I got ~160MB/sec throughput. In a OpenVZ container (we've been using openvz for everything since I thought it had less overhead) I got ~15MB/sec. There was nothing else running at the time.

So, OBVIOUSLY that's a problem. The first step to solving any problem is to make sure you are up-to-date. I did an apt-get update;apt-get upgrade and nothing was updated. So the next step, I went to the website and noticed a new release was made recently - I downloaded that and re-installed from scratch.

The new install is now running 2.6.32-5 kernel. Now, I haven't gotten to far into my testing on the new server yet, but I've noticed some strange results and was wondering what the best way to test for hard drive throughput is?

(EDIT: The new ProxMox using 2.6.32-5 kernel seems to give a much better performance than the 2.6.18-4 kernel did, but see below)

By strange, I mean I am seeing things like buffered reads (from pveperf and hdparm) show ~89MB/sec on the single hitachi drive (I haven't even gotten to the raid yet, I'm trying to create a big comparison matrix) under the host, but 160 MB/sec to 380 MB/sec under a VM... And the numbers in a VM can vary by as much as 100 MB/sec from one run to the next!

So far, my matrix includes dbench, bonnie++, hdparm, and just an good ole fashion dd if=/dev/zero of=blah bs=8k count=256k to measure speed.

The VM's only have 512MB of RAM allocated, and I'm trying to find what has the best performance...

OpenVZ or KVM

And then what options should I use: IDE or SCSI, LVM or NO LVM, RAW, QCOW, or VMDK?

If someone already has all of these answers, then please enlighten me.

 

[screenie]

And then what options should I use: IDE or SCSI, LVM or NO LVM, RAW, QCOW, or VMDK?

If someone already has all of these answers, then please enlighten me.

VIRTIO, LVM and RAW for more data security or QCOW2 because of the features....

 

[oeginc]

VIRTIO, LVM and RAW for more data security or QCOW2 because of the features....

What features (other than snapshot, and growing disk space) does qcow2 have? I checked Wikipedia but it wasn't very helpful..

 

[screenie]

What features (other than snapshot, and growing disk space) does qcow2 have? I checked Wikipedia but it wasn't very helpful..

compression, encryption and what you probably not want in an production environment - write cache;
raw can be restored to iscsi or lvm - qcow2 only on a fileystem like nfs, also qcow2 is slower then raw;
if you want a encrypted or compressed image you need to use qcow2 - raw gives you more data security after a crash because there is no cache and all data are written to the disk directly;

 

[oeginc]

compression, encryption and what you probably not want in an production environment - write cache;
raw can be restored to iscsi or lvm - qcow2 only on a fileystem like nfs, also qcow2 is slower then raw;
if you want a encrypted or compressed image you need to use qcow2 - raw gives you more data security after a crash because there is no cache and all data are written to the disk directly;

So are we saying that OpenVZ itself isn't a viable option? We've been running it because we have 8GB of RAM and a bunch of servers (MySQL, Apache, etc) that take advantage of the shared RAM space. To the best of my knowledge, KVM needs to hard-chop the RAM and it can't expand/detract dynamically, correct?

 

[screenie]

So are we saying that OpenVZ itself isn't a viable option? We've been running it because we have 8GB of RAM and a bunch of servers (MySQL, Apache, etc) that take advantage of the shared RAM space. To the best of my knowledge, KVM needs to hard-chop the RAM and it can't expand/detract dynamically, correct?

Correct, for KVM ther is a feature called KSM which can save memory when you host more vm's with the same os or applications on it;
But you need to check if your kernel support this - see the release notes for the PVE kernels;

There is no real 'shared Ram space' - the vm get the memory you assign, if you assign more memory than you physically have it is overcommitting;

If you have 8gb memory you need to make sure not overcommitting them on KVM or OpenVZ anyway, maybe in a test environment but never on production environments;
Your vm's will crash if the try to assign more memory which is not availiable....

OpenVZ uses the same filesystem from the host and only supports ext3 at the moment, so possibly native speed to the disks with no layer in between;

 

[oeginc]

Correct, for KVM ther is a feature called KSM which can save memory when you host more vm's with the same os or applications on it;
But you need to check if your kernel support this - see the release notes for the PVE kernels;

There is no real 'shared Ram space' - the vm get the memory you assign, if you assign more memory than you physically have it is overcommitting;

If you have 8gb memory you need to make sure not overcommitting them on KVM or OpenVZ anyway, maybe in a test environment but never on production environments;
Your vm's will crash if the try to assign more memory which is not availiable....

OpenVZ uses the same filesystem from the host and only supports ext3 at the moment, so possibly native speed to the disks with no layer in between;

How do you implement KSM? I've checked my default install of the latest ProxMox build and it doesn't seem to be enabled in the kernel. Is there something that supports OpenVZ -AND- KVM w/KSM?

And it's not so much an issue of overcommitting (which I am doing with OpenVZ because it never gets used simultaneously, at least in my situation) as much as memory management. For example, under OpenVZ - I have numerous hosts that are only using 39MB - 117MB of RAM - Those same hosts under KVM would require AT LEAST 512MB to run at the same efficiency.

Does any other KVM type system out there provide better memory management (Ie: Xen, VMWare, etc?) or are they all pretty much the same?

 

[tom]

KSM does not work on any OpenVZ enabled Kernel. If you want KSM, you need a KVM only kernel (our 2.6.35 branch).

and no, there is no better solutions than Proxmox VE available - did you expect another answer from me ?

 

[oeginc]

Can anyone explain why pveperf, hdparm, and bonnie all show a significant improvement in throughput under a container vs. the host?

For example, I am seeing...

Host PVEPerf Buf. Reads @ 87.84 MB/sec, Guests from 186-346 MB/sec
Host hdParm Buf. Reads @ 87.60 MB/sec, Guests from 278-367 MB/sec
Host bonnie++ Rnd. Seeks @ 162/sec, Guests from 2604-3088/sec

But some other tests, I get what I expect (or some resemblance there of):

Host pveperf fsyncs @ 958.25/sec, Guests from 429-606
Host dbench throughput @ 88.35MB/sec, Guests from 67.55-87.89MB/sec
Host hdparm cached reads @ 2615MB/sec, Guests from 1429-1988MB/sec
Host dd @ 215MB/sec, Guests from 64-160MB/sec

What would be the best way to test simulated "real-world" usage (Ie: database server, mail server, webserver scenarios)? How do you test your disk & network throughput under various scenarios?

And please don't say setting up servers and testing them in the real world..

 

[oeginc]

Nobody knows of a good way to test HD performance? Preferrably local or NAS drives...

 

[deepc]

Would this help you?

http://www.linux.com/archive/feature/131063

 

[tom]

take a look at phoronix test suite.
http://www.phoronix-test-suite.com/