Poor hard drive performance...

oeginc

Member
Mar 21, 2009
133
0
16
We had a server crash recently (power supply died, turns out we were running 150 watt power supplies in our 3U rackmount servers and they just couldn't keep up with 6 hard drives, raid card, etc... Surprised it worked at all! LOL!)....

Anyways, we've been having some big problems keeping up with disk I/O, so I thought while the server was down I'd play around a little bit.

I don't have the exact PVE version information, but the server (as it stood) was originally a v1.5 install that had been apt-get upgrade'd to v1.6 and was running a 2.6.18-4 kernel.

The hardware:
Gigabyte motherboard
AMD Phenom x4 9850 quad-core processor
8GB RAM
Highpoint 2640x4 RAID Controller card (I know, I have adaptec 5805z's here just waiting to be swapped out).
1 Hitachi 500GB SATA2 7,200 RPM Drive - Boot & Proxmox
1 Western Digital Black 2TB SATA2 7,200 RPM Drive - VM Backups
4 Segate 500GB SATA2 7,2000 RPM Drives in RAID 10 - Virtual Machines

In the host running dbench on the RAID10 I got ~160MB/sec throughput. In a OpenVZ container (we've been using openvz for everything since I thought it had less overhead) I got ~15MB/sec. There was nothing else running at the time.

So, OBVIOUSLY that's a problem. The first step to solving any problem is to make sure you are up-to-date. I did an apt-get update;apt-get upgrade and nothing was updated. So the next step, I went to the website and noticed a new release was made recently - I downloaded that and re-installed from scratch.

The new install is now running 2.6.32-5 kernel. Now, I haven't gotten to far into my testing on the new server yet, but I've noticed some strange results and was wondering what the best way to test for hard drive throughput is?

(EDIT: The new ProxMox using 2.6.32-5 kernel seems to give a much better performance than the 2.6.18-4 kernel did, but see below)

By strange, I mean I am seeing things like buffered reads (from pveperf and hdparm) show ~89MB/sec on the single hitachi drive (I haven't even gotten to the raid yet, I'm trying to create a big comparison matrix) under the host, but 160 MB/sec to 380 MB/sec under a VM... And the numbers in a VM can vary by as much as 100 MB/sec from one run to the next!

So far, my matrix includes dbench, bonnie++, hdparm, and just an good ole fashion dd if=/dev/zero of=blah bs=8k count=256k to measure speed.

The VM's only have 512MB of RAM allocated, and I'm trying to find what has the best performance...

OpenVZ or KVM

And then what options should I use: IDE or SCSI, LVM or NO LVM, RAW, QCOW, or VMDK?

If someone already has all of these answers, then please enlighten me. ;)
 
Last edited:

screenie

Active Member
Jul 21, 2009
157
4
38
And then what options should I use: IDE or SCSI, LVM or NO LVM, RAW, QCOW, or VMDK?

If someone already has all of these answers, then please enlighten me. ;)
VIRTIO, LVM and RAW for more data security or QCOW2 because of the features....
 

oeginc

Member
Mar 21, 2009
133
0
16
VIRTIO, LVM and RAW for more data security or QCOW2 because of the features....
What features (other than snapshot, and growing disk space) does qcow2 have? I checked Wikipedia but it wasn't very helpful..
 

screenie

Active Member
Jul 21, 2009
157
4
38
What features (other than snapshot, and growing disk space) does qcow2 have? I checked Wikipedia but it wasn't very helpful..
compression, encryption and what you probably not want in an production environment - write cache;
raw can be restored to iscsi or lvm - qcow2 only on a fileystem like nfs, also qcow2 is slower then raw;
if you want a encrypted or compressed image you need to use qcow2 - raw gives you more data security after a crash because there is no cache and all data are written to the disk directly;
 

oeginc

Member
Mar 21, 2009
133
0
16
compression, encryption and what you probably not want in an production environment - write cache;
raw can be restored to iscsi or lvm - qcow2 only on a fileystem like nfs, also qcow2 is slower then raw;
if you want a encrypted or compressed image you need to use qcow2 - raw gives you more data security after a crash because there is no cache and all data are written to the disk directly;
So are we saying that OpenVZ itself isn't a viable option? We've been running it because we have 8GB of RAM and a bunch of servers (MySQL, Apache, etc) that take advantage of the shared RAM space. To the best of my knowledge, KVM needs to hard-chop the RAM and it can't expand/detract dynamically, correct?
 

screenie

Active Member
Jul 21, 2009
157
4
38
So are we saying that OpenVZ itself isn't a viable option? We've been running it because we have 8GB of RAM and a bunch of servers (MySQL, Apache, etc) that take advantage of the shared RAM space. To the best of my knowledge, KVM needs to hard-chop the RAM and it can't expand/detract dynamically, correct?
Correct, for KVM ther is a feature called KSM which can save memory when you host more vm's with the same os or applications on it;
But you need to check if your kernel support this - see the release notes for the PVE kernels;

There is no real 'shared Ram space' - the vm get the memory you assign, if you assign more memory than you physically have it is overcommitting;

If you have 8gb memory you need to make sure not overcommitting them on KVM or OpenVZ anyway, maybe in a test environment but never on production environments;
Your vm's will crash if the try to assign more memory which is not availiable....

OpenVZ uses the same filesystem from the host and only supports ext3 at the moment, so possibly native speed to the disks with no layer in between;
 

oeginc

Member
Mar 21, 2009
133
0
16
Correct, for KVM ther is a feature called KSM which can save memory when you host more vm's with the same os or applications on it;
But you need to check if your kernel support this - see the release notes for the PVE kernels;

There is no real 'shared Ram space' - the vm get the memory you assign, if you assign more memory than you physically have it is overcommitting;

If you have 8gb memory you need to make sure not overcommitting them on KVM or OpenVZ anyway, maybe in a test environment but never on production environments;
Your vm's will crash if the try to assign more memory which is not availiable....

OpenVZ uses the same filesystem from the host and only supports ext3 at the moment, so possibly native speed to the disks with no layer in between;
How do you implement KSM? I've checked my default install of the latest ProxMox build and it doesn't seem to be enabled in the kernel. Is there something that supports OpenVZ -AND- KVM w/KSM?

And it's not so much an issue of overcommitting (which I am doing with OpenVZ because it never gets used simultaneously, at least in my situation) as much as memory management. For example, under OpenVZ - I have numerous hosts that are only using 39MB - 117MB of RAM - Those same hosts under KVM would require AT LEAST 512MB to run at the same efficiency.

Does any other KVM type system out there provide better memory management (Ie: Xen, VMWare, etc?) or are they all pretty much the same?
 

tom

Proxmox Staff Member
Staff member
Aug 29, 2006
14,512
587
133
KSM does not work on any OpenVZ enabled Kernel. If you want KSM, you need a KVM only kernel (our 2.6.35 branch).

and no, there is no better solutions than Proxmox VE available - did you expect another answer from me :)?
 

oeginc

Member
Mar 21, 2009
133
0
16
Can anyone explain why pveperf, hdparm, and bonnie all show a significant improvement in throughput under a container vs. the host?

For example, I am seeing...

Host PVEPerf Buf. Reads @ 87.84 MB/sec, Guests from 186-346 MB/sec
Host hdParm Buf. Reads @ 87.60 MB/sec, Guests from 278-367 MB/sec
Host bonnie++ Rnd. Seeks @ 162/sec, Guests from 2604-3088/sec

But some other tests, I get what I expect (or some resemblance there of):

Host pveperf fsyncs @ 958.25/sec, Guests from 429-606
Host dbench throughput @ 88.35MB/sec, Guests from 67.55-87.89MB/sec
Host hdparm cached reads @ 2615MB/sec, Guests from 1429-1988MB/sec
Host dd @ 215MB/sec, Guests from 64-160MB/sec

What would be the best way to test simulated "real-world" usage (Ie: database server, mail server, webserver scenarios)? How do you test your disk & network throughput under various scenarios?

And please don't say setting up servers and testing them in the real world.. ;)
 

oeginc

Member
Mar 21, 2009
133
0
16
Nobody knows of a good way to test HD performance? Preferrably local or NAS drives...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!