Disk I/O Benchmarks of Proxmox / KVM Disk Types

[Charlie]

Which is faster disk type? QCOW2, RAW, VMDK? IDE or VirtIO? Cache or No Cache?

In order to identify the best disk type and bus method, we needed some rudimentary benchmarks. We've previously added some of this data in another thread; however, this has additional information and details, and is KVM (Proxmox) specific. There will be additional benchmark on their CPU usage coming soon.

Here are the KVM Disk Types utilized by Proxmox VE - http://www.imagehousing.com/image/813009. On the right column for each disk type, you will see a NO CACHE option. This means the flag CACHE=NONE was used for that particular configuration. To our surprise, the VMDK format with VirtIO was the overall "faster" setup in this test. This thread has more details on the subject on Caching - http://arstechnica.com/civis/viewtopic.php?f=16&t=1143694

The tests were performed on guests machines running Windows 2008 R2 SP1 x64. Each test configuration were performed on a clean / unattended setup (standard configuration) of the Operating System. Each OS has 1256 MB of RAM allocated. The hardware used was a low-end Dell Precision 2.0 GHz dual-core with 7200 RPM 500 GB Seagate Barracuda Drive. We intend to repeat these tests on newer server class machine by Intel when a spare becomes available. We would also like to run these test all concurrently; however, more resources are required.

Optional Link to the tests - http://i51.tinypic.com/158bcl4.gif

 

[dietmar]

How did you make sure that your images are at the same disk location? Disk speed is highly dependent on sector location.

 

[Charlie]

The best way this was ensured was each image / test was located on /VM test partition of 40 GB. Even taking that into consideration, it will not be bit by bit exact, but close enough for our test. This was for our benefit and we share only the results from our independent findings.

 

[udo]

Which is faster disk type? QCOW2, RAW, VMDK? IDE or VirtIO? Cache or No Cache?
... The hardware used was a low-end Dell Precision 2.0 GHz dual-core with 7200 RPM 500 GB Seagate Barracuda Drive.

Hi,
sorry that i say this, but why you waste your time with such benchmark? In your setup the bottleneck is the Sata-Drive. All good values depends on caching and are not very usefull (IMHO).
For good IO-Power in an virtualisation setup you need good io-performance. This can't be reached with a single sata-drive...
The fasted way should be lvm-storage (raw to the disk-lv). No filesystemlayer between.

Udo

 

[Charlie]

True; however, we have two types of client. The SMB market and larger firms. In alot of instance, budget is a factor. So with a limited budget, we still need to find the best value (including configurations) for the client. Another example, we may look utilize pfSense firewall solution for SMB market vs a Juniper/Checkpoint for another. We try to find the best solution base on budget and product feature requirements.

If we had our choice, it would be Enterprise grade Solid State PCI-Express drives, but not all businesses can afford that. And we purposedly tested, the lowest and most limited end machine we could in order to see the "base" we could work with at those levels. Thus, for our purpose, it was not wasted time.

 

[udo]

...
If we had our choice, it would be Enterprise grade Solid State PCI-Express drives, but not all businesses can afford that...

Hi Charlie,
perhaps you have the chance to test the OCZ Vertex for your setup (120GB is not very large, of course). It's not enterprise grade, but i have good performance experienses with such a disk.
At this time i have tested the Vertex 2 only but next week i can do some tests with the Vertex 3 model.

You should also try to use lvm-storage on the single disk for test. But in a single-disk setup you can only reach that with handwork - the normal installer use the full disk for the lvm-volumegroup pve.

Udo

 

[zc1974]

Which is faster disk type? QCOW2, RAW, VMDK? IDE or VirtIO? Cache or No Cache?

Here are the KVM Disk Types utilized by Proxmox VE - http://www.imagehousing.com/image/813009. On the right column for each disk type, you will see a NO CACHE option. This means the flag CACHE=NONE was used for that particular configuration. To our surprise, the VMDK format with VirtIO was the overall "faster" setup in this test.

Confirmed VMDK performed best.

 

[docent]

Hi Charlie,
Unfortunately I can't get so high values even on the better HW (i5-2400 @ 3.1GHz, SATA6G, WD5002AALX).
-----------------------------------------------------------------------
CrystalDiskMark 3.0 B5 x64 (C) 2007-2010 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/sec [SATA/300 = 300,000,000 bytes/sec]

Sequential Read : 105.236 MB/s
Sequential Write : 12.530 MB/s
Random Read 512KB : 44.229 MB/s
Random Write 512KB : 12.552 MB/s
Random Read 4KB (QD=1) : 0.629 MB/s [ 153.6 IOPS]
Random Write 4KB (QD=1) : 0.455 MB/s [ 111.2 IOPS]
Random Read 4KB (QD=32) : 0.719 MB/s [ 175.6 IOPS]
Random Write 4KB (QD=32) : 0.503 MB/s [ 122.7 IOPS]

Test : 1000 MB [C: 32.7% (10.4/31.9 GB)] (x5)
Date : 2011/07/14 16:42:49
OS : Windows Server 2008 R2 Server Standard Edition (full installation) [6.1 Build 7600] (x64)
cache=none, ide

-----------------------------------------------------------------------
Sequential Read : 109.730 MB/s
Sequential Write : 7.031 MB/s
Random Read 512KB : 39.069 MB/s
Random Write 512KB : 7.048 MB/s
Random Read 4KB (QD=1) : 0.643 MB/s [ 157.0 IOPS]
Random Write 4KB (QD=1) : 0.473 MB/s [ 115.4 IOPS]
Random Read 4KB (QD=32) : 1.758 MB/s [ 429.2 IOPS]
Random Write 4KB (QD=32) : 0.466 MB/s [ 113.7 IOPS]

Test : 1000 MB [D: 0.8% (69.0/8189.0 MB)] (x5)
Date : 2011/07/14 17:21:49
OS : Windows Server 2008 R2 Server Standard Edition (full installation) [6.1 Build 7600] (x64)
cache=none, virtio

-----------------------------------------------------------------------
Sequential Read : 563.751 MB/s
Sequential Write : 100.757 MB/s
Random Read 512KB : 571.188 MB/s
Random Write 512KB : 85.092 MB/s
Random Read 4KB (QD=1) : 23.016 MB/s [ 5619.2 IOPS]
Random Write 4KB (QD=1) : 1.067 MB/s [ 260.5 IOPS]
Random Read 4KB (QD=32) : 29.012 MB/s [ 7083.1 IOPS]
Random Write 4KB (QD=32) : 1.073 MB/s [ 262.0 IOPS]

Test : 1000 MB [C: 32.6% (10.4/31.9 GB)] (x5)
Date : 2011/07/14 18:04:47
OS : Windows Server 2008 R2 Server Standard Edition (full installation) [6.1 Build 7600] (x64)
cache=writeback, ide


Every time, when I try to test the speed of the VIRTIO disk with 'cache=writeback', the virtual machine hangs.


And what about a CPU load? When I test the speed of VIRTIO HDD, I a have very high load of the CPU, about 30-60%.

 

[Charlie]

The tests were updated with other hypervisors - http://accends.wordpress.com/

 

[Charlie]

After further testing, we need to state the disk i/o test results above are ONLY for the Proxmox 1.8 version using kernel 2.6.32-4-pve.

This can significantly change upon changes in kernel or versions. For example, in the 1.8 release with kernel 2.6.32-4-pve, we found VMDK / IDE has excellent results; however, that is not the case with the current Proxmox release kernel 2.6.35-1-pve. See below.

http://accends.files.wordpress.com/2011/08/vmdkperf-kerneldifference.gif





Whenever you upgrade and/or go into production, make sure you test for the change as this was a significant change.

 

[alchemycs]

I am in the midst of I/O testing various setting for Proxmox VM options, and even though this thread is a bit old, I thought I should contribute some of what I have done.

I am testing on a Dell Poweredge 2950 w/ 2x quad core Intel CPUs @ 2.67GHz, 24GB of 667MHz ram, and 6x 10k SAS drives in a RAID-5 on a Perc5 card running Proxmox 1.9 with CentOS 5 VM guests.

I initially found Proxmox as an alternative to a strictly OpenVZ setup (which I really liked for its insanely low overhead) so I was excited to see a pre-built option which included fully virtualized as well as OpenVZ.

I became even more excited when I read about the automatic fail-over of VMs in a cluster with DRBD, but became disheartened when I realised it was only for KVM guests, but I am interested enough that I wanted to see how much performance is lost when using KVM instead of OpenVZ.

I ran 5 sets of bonnie++ 1.03e on a (roughly) similar location on the harddrive for the physical host and each of the KVM IO options, IDE, SCSI and virtio, as well as OpenVZ.
To eliminate possible memory caching (which I think skews your data, Charlie) I turned off swap and created a 22.75GB memory file system and filled it (leaving 1.25GB effective memory (I set each VM to have 1GB of memory)).

Here is basically what I found:

Version	1.03e	------Sequential Output------	--Sequential Input-	--Random-
		-Per Chr-	--Block--	-Rewrite-	-Per Chr-	--Block--	--Seeks--
Machine	Size	K/sec	%CP	K/sec	%CP	K/sec	%CP	K/sec	%CP	K/sec	%CP	/sec	%CP
bare machine	4G	65268.2	96.6	298549.4	48	110636.6	17.6	64705.6	88.2	268427.2	22.6	851.5	1.2
virtio	4G	63825.4	98	200290.2	64.4	51144.4	15.6	64732.8	91	207052.2	30.2	683.96	3.2
% worse from bare metal	0.98	1.01	0.67	1.34	0.46	0.89	1.00	1.03	0.77	1.34	0.80	2.67
SCSI	4G	60039.8	97.8	197433.2	64.8	52513	12.4	64079.8	92.2	214331.6	23.6	628.52	4
% worse from bare metal	0.92	1.01	0.66	1.35	0.47	0.70	0.99	1.05	0.80	1.04	0.74	3.33
IDE	4G	59577.2	97.2	210090.6	74	48560.8	10	60905	90.6	123826.2	8.4	270.54	3.2
% worse from bare metal	0.91	1.01	0.70	1.54	0.44	0.57	0.94	1.03	0.46	0.37	0.32	2.67
OpenVZ	4G	71408.6	95.6	298347.2	48.2	92573.2	14.4	64182.4	81.4	292892.4	21.4	816.92	1.2
% worse from bare metal	1.09	0.99	1.00	1.00	0.84	0.82	0.99	0.92	1.09	0.95	0.96	1.00

OpenVZ does great with IO; at worse it was 84% as fast as rewriting files and its CPU usage is about the same as the bare metal.
The virtio does a pretty good job at reading files, but it almost half as slow at updating files as OpenVZ and it uses a lot more CPU doing everything, especially random seeks.
the SCSI emulation is pretty comparable to the virtio, but the IDE is definite much farther behind.

This isn't a huge dataset, and it's by no means done, but I just thought I'd toss this out in case it was of interest.
I will be writing a blog post on my website about this soon & can link to it when I'm done if there is interest.

Thank you!

 

[Pedro Roger M Vasconcelos]

I am in the midst of I/O testing various setting for Proxmox VM options, and even though this thread is a bit old, I thought I should contribute some of what I have done.

I am testing on a Dell Poweredge 2950 w/ 2x quad core Intel CPUs @ 2.67GHz, 24GB of 667MHz ram, and 6x 10k SAS drives in a RAID-5 on a Perc5 card running Proxmox 1.9 with CentOS 5 VM guests.

I initially found Proxmox as an alternative to a strictly OpenVZ setup (which I really liked for its insanely low overhead) so I was excited to see a pre-built option which included fully virtualized as well as OpenVZ.

I became even more excited when I read about the automatic fail-over of VMs in a cluster with DRBD, but became disheartened when I realised it was only for KVM guests, but I am interested enough that I wanted to see how much performance is lost when using KVM instead of OpenVZ.

I ran 5 sets of bonnie++ 1.03e on a (roughly) similar location on the harddrive for the physical host and each of the KVM IO options, IDE, SCSI and virtio, as well as OpenVZ.
To eliminate possible memory caching (which I think skews your data, Charlie) I turned off swap and created a 22.75GB memory file system and filled it (leaving 1.25GB effective memory (I set each VM to have 1GB of memory)).

Here is basically what I found:

Version	1.03e	------Sequential Output------	--Sequential Input-	--Random-
		-Per Chr-	--Block--	-Rewrite-	-Per Chr-	--Block--	--Seeks--
Machine	Size	K/sec	%CP	K/sec	%CP	K/sec	%CP	K/sec	%CP	K/sec	%CP	/sec	%CP
bare machine	4G	65268.2	96.6	298549.4	48	110636.6	17.6	64705.6	88.2	268427.2	22.6	851.5	1.2
virtio	4G	63825.4	98	200290.2	64.4	51144.4	15.6	64732.8	91	207052.2	30.2	683.96	3.2
% worse from bare metal	0.98	1.01	0.67	1.34	0.46	0.89	1.00	1.03	0.77	1.34	0.80	2.67
SCSI	4G	60039.8	97.8	197433.2	64.8	52513	12.4	64079.8	92.2	214331.6	23.6	628.52	4
% worse from bare metal	0.92	1.01	0.66	1.35	0.47	0.70	0.99	1.05	0.80	1.04	0.74	3.33
IDE	4G	59577.2	97.2	210090.6	74	48560.8	10	60905	90.6	123826.2	8.4	270.54	3.2
% worse from bare metal	0.91	1.01	0.70	1.54	0.44	0.57	0.94	1.03	0.46	0.37	0.32	2.67
OpenVZ	4G	71408.6	95.6	298347.2	48.2	92573.2	14.4	64182.4	81.4	292892.4	21.4	816.92	1.2
% worse from bare metal	1.09	0.99	1.00	1.00	0.84	0.82	0.99	0.92	1.09	0.95	0.96	1.00

OpenVZ does great with IO; at worse it was 84% as fast as rewriting files and its CPU usage is about the same as the bare metal.
The virtio does a pretty good job at reading files, but it almost half as slow at updating files as OpenVZ and it uses a lot more CPU doing everything, especially random seeks.
the SCSI emulation is pretty comparable to the virtio, but the IDE is definite much farther behind.

This isn't a huge dataset, and it's by no means done, but I just thought I'd toss this out in case it was of interest.
I will be writing a blog post on my website about this soon & can link to it when I'm done if there is interest.

Thank you!

alchemycs,

what was your command line for bonnie++ in these benchmarks?

 

[alchemycs]

alchemycs,

what was your command line for bonnie++ in these benchmarks?

I believe it was just bonnie++ -u0 -s4096
I ran it as the root user, so I had to specify -u and a UID to run as.