Disk Performance of KVM Virtual machine....

abadger

New Member
Sep 16, 2012
16
0
1
On a Linux (Ubuntu 12.04) server if you cp the contents of /usr/bin (approx 200M) it will take approx less than 5 seconds.E.G:-

# du -s -h /usr/bin
215M /usr/bin

# time cp -r /usr/bin/ home

real 0m2.174s
user 0m0.004s
sys 0m0.448s


On a Ubuntu server that is a KVM guest machine, the same command takes approx 20 seconds.

virt-server # time cp -r /usr/bin/ home

real 0m20.076s
user 0m0.008s
sys 0m0.484s


Firstly, can someone verify they see similar results on their Virtual machines. I have used Virtio disk drivers...and am using qcow2 disk based images.

I know you can use OpenVZ for Ubuntu guests, but my real issue is to improve disk performance on Windows 2003 guest machines, so any advice on how to speed this up.....

I tried a raw image rather than qcow2...this made very little difference. Would using Volumes make a bigger difference ?

Thanks Dave
 
Last edited:
Host node 01
10/21/12 /dev/sda:
Hours:
SMART Errors: 0
Reallocated / Pending : /
Read Speed: 402 MB/s

linux vm same node
Serial No: QM00001
10/21/12 /dev/sda:
Hours: 1
SMART Errors: 0
Reallocated / Pending : 0 /
Read Speed: 207 MB/s


Half the speed and this is a Raw image
Their is a speed difference i just never thought it would be half the speed.
 
...and how did you get this numbers? What tool+parameters+vm configuration did you use?
 
This vm 100.conf file.
boot: cdn
bootdisk: ide0
cores: 2
cpu: core2duo
ide0: local:100/vm-100-disk-1.raw
ide2: local:iso/CentOS-6.0-x86_64-bin-DVD1.iso,media=cdrom,size=4139454K
memory: 4096
name: Virtualmin
net0: virtio=52:A2:E5:C4:0C:FD,bridge=vmbr0
ostype: l26
sockets: 1

2 ways i came to these numbers

1: hdparm -Tt /dev/sda
ran this a 6 times got the average speed

2: smartctr you can use
here is the scripts that was used to get the speed test using smartctl



#!/bin/bash#First let's determine the deviceif [ $# -gt 0 ]; then if [ -b $1 ]; then DRIVE=$1 else echo "$1 is not a block device? Try running fdisk -l for a list of drives" fielse #Try hda? if [ -b /dev/hda ]; then DRIVE=/dev/hda elif [ -b /dev/sda ]; then DRIVE=/dev/sda else echo Could not autofind a drive to work with. Try designating one: smarttest /dev/sdc exit fifi#echo "Turning on smart"smartctl -s on $DRIVE >/dev/null 2>&1#if [ $? -ne 0 ]; then# echo Turning SMART on the drive failed for some reason# exit#fi#echo "Running Quick Test"smartctl --test short $DRIVE >/dev/null 2>&1sleep 2mHOURS=`smartctl -d ata -a $DRIVE | grep "Power_On_Hours" | tr " " "\n" | tail -n 1`if [ "$HOURS" == "" ]; then HOURS=`smartctl -d ata -a $DRIVE | grep "# 1" | cut -c 64-75`fiSERIALNUMBER=`smartctl -d ata -a $DRIVE | grep "Serial" | head -n 1 | awk '{print $3}'`ERRORCOUNT=`smartctl -d ata -a $DRIVE | grep "Error" | grep "occurred" | head -n 1 | cut -f 2 -d " "`ERRORTIME=`smartctl -d ata -a $DRIVE | grep "Error" | grep "occurred" | head -n 1 | cut -f 8 -d " "`if [ "$ERRORCOUNT" = "" ] ; thenERRORCOUNT=0ERRORTIME=0fiREALLOCATEDSECTORS=`smartctl -d ata -a $DRIVE | grep "Reallocated_Sector" | tr " " "\n" | tail -n 1`SPEED=`hdparm -t $DRIVE | tr " " "\n" | tail -n 2 | head -n 1 | cut -f 1 -d .`PENDINGSECTORS=`smartctl -d ata -a $DRIVE | grep "Current_Pending_Sector" | tr " " "\n" | tail -n 1`echo -e "\033[1;42;37mTest Complete\033[0m"echo "Serial No: $SERIALNUMBER"echo "`date +%D` $DRIVE:"echo "Hours: $HOURS "echo -n "SMART Errors: $ERRORCOUNT"if [ $ERRORCOUNT -gt 0 ]; thenlet WHENITHAPPEND=$HOURS-$ERRORTIMEecho "(last $WHENITHAPPEND hours ago)"elseechofiecho "Reallocated / Pending : $REALLOCATEDSECTORS / $PENDINGSECTORS"echo "Read Speed: $SPEED MB/s"echoechoif [ $HOURS -ge 20000 ]; thenecho -e '\033[5;1;37;41mWARNING:\033[0m This drive has over 20,000 hours on it and should not be used as a Primary'fiif [ $ERRORCOUNT -gt 0 ]; thenecho -e "\033[5;1;37;41mWARNING:\033[0m The last smart error was $WHENITHAPPEND hours ago, use your judgement to tell if this is relevant (use smartctl -a $DRIVE for more info)"fiif [ $REALLOCATEDSECTORS -gt 0 ]; thenecho -e '\033[5;1;37;41mWARNING:\033[0m This drive has some reallocated sectors, this should not be used as a primary and requires judgement if it is to be used for a secondary'fiif [ $PENDINGSECTORS -gt 0 ]; thenecho -e '\033[5;1;37;41mWARNING:\033[0m This drive has some pending sectors, this shouldn not be used as a primary and requires judgement if it is to be used for a secondary'fiif [ $SPEED -le 25 ]; thenecho -e "\033[5;1;37;41mWARNING:\033[0m This drive is only reading at $SPEED MB/s, which seems slow, is this drive ok and is it on a good motherboard and ide cable?"fi

scripts doesn't read correctly in post let me know if you understand it otherwise i will post a link if that is ok.
 
Last edited:
It's a wiki, use the related tags, that, AFARI, are \[code ] and \[/code] before and after the text regarding code (without the '\' that I had to use to escape the tag in this message).
 
qcow2 can be slow (as they are metadatas to be updated in the qcow2 file)

But you shouldn't have soo much differences with raw file.

first: use virtio
then:
Maybe this is a writeback problem. Do you have a raid hardware controller with some cache ?

Also what is your /etc/fstab setting and kernel version ? (host and guest)
 
These are the new results for the test same hardware just running virtio for the drive instead of ide

Host-01

Serial No:
10/22/12 /dev/dm-2:
Hours:
SMART Errors: 0
Reallocated / Pending : /
Read Speed: 352 MB/s


hdparm -Tt /dev/dm-2
/dev/dm-2:
Timing cached reads: 14832 MB in 2.00 seconds = 7423.62 MB/sec
Timing buffered disk reads: 1118 MB in 3.00 seconds = 372.20 MB/sec


hdparm -Tt /dev/dm-2
/dev/dm-2:
Timing cached reads: 14666 MB in 2.00 seconds = 7340.38 MB/sec
Timing buffered disk reads: 1220 MB in 3.01 seconds = 405.96 MB/sec


This is with hdparm running on both the vm and host at same time
hdparm -Tt /dev/dm-2
/dev/dm-2:
Timing cached reads: 14432 MB in 2.00 seconds = 7223.19 MB/sec
Timing buffered disk reads: 1122 MB in 3.02 seconds = 371.77 MB/sec

VM with virto for the drive
sh smart.sh /dev/vda1
Test Complete
Serial No:
10/22/12 /dev/vda1:
Hours:
SMART Errors: 0
Reallocated / Pending : /
Read Speed: 360 MB/s




hdparm -Tt /dev/vda1


/dev/vda1:
Timing cached reads: 13570 MB in 2.00 seconds = 6795.83 MB/sec
Timing buffered disk reads: 500 MB in 0.98 seconds = 509.13 MB/sec
[root@localhost ~]# hdparm -Tt /dev/vda1

hdparm -Tt /dev/vda1


This is with hdparm running on both the vm and host at same time
/dev/vda1:
Timing cached reads: 14058 MB in 2.00 seconds = 7040.09 MB/sec
Timing buffered disk reads: 500 MB in 1.92 seconds = 259.97 MB/sec


Code for Drive script



\[/code]
#!/bin/bash
#First let's determine the device
if [ $# -gt 0 ]; then
if [ -b $1 ]; then
DRIVE=$1
else
echo "$1 is not a block device? Try running fdisk -l for a list of drives"
fi
else
#Try hda?
if [ -b /dev/hda ]; then
DRIVE=/dev/hda
elif [ -b /dev/sda ]; then
DRIVE=/dev/sda
else
echo Could not autofind a drive to work with. Try designating one: smarttest /dev/sdc
exit
fi
fi

#echo "Turning on smart"
smartctl -s on $DRIVE >/dev/null 2>&1
#if [ $? -ne 0 ]; then
# echo Turning SMART on the drive failed for some reason
# exit
#fi

#echo "Running Quick Test"
smartctl --test short $DRIVE >/dev/null 2>&1
sleep 2m
HOURS=`smartctl -d ata -a $DRIVE | grep "Power_On_Hours" | tr " " "\n" | tail -n 1`

if [ "$HOURS" == "" ]; then
HOURS=`smartctl -d ata -a $DRIVE | grep "# 1" | cut -c 64-75`
fi

SERIALNUMBER=`smartctl -d ata -a $DRIVE | grep "Serial" | head -n 1 | awk '{print $3}'`

ERRORCOUNT=`smartctl -d ata -a $DRIVE | grep "Error" | grep "occurred" | head -n 1 | cut -f 2 -d " "`
ERRORTIME=`smartctl -d ata -a $DRIVE | grep "Error" | grep "occurred" | head -n 1 | cut -f 8 -d " "`

if [ "$ERRORCOUNT" = "" ] ; then
ERRORCOUNT=0
ERRORTIME=0
fi
REALLOCATEDSECTORS=`smartctl -d ata -a $DRIVE | grep "Reallocated_Sector" | tr " " "\n" | tail -n 1`
SPEED=`hdparm -t $DRIVE | tr " " "\n" | tail -n 2 | head -n 1 | cut -f 1 -d .`

PENDINGSECTORS=`smartctl -d ata -a $DRIVE | grep "Current_Pending_Sector" | tr " " "\n" | tail -n 1`

echo -e "\033[1;42;37mTest Complete\033[0m"
echo "Serial No: $SERIALNUMBER"
echo "`date +%D` $DRIVE:"
echo "Hours: $HOURS "
echo -n "SMART Errors: $ERRORCOUNT"
if [ $ERRORCOUNT -gt 0 ]; then
let WHENITHAPPEND=$HOURS-$ERRORTIME
echo "(last $WHENITHAPPEND hours ago)"
else
echo
fi
echo "Reallocated / Pending : $REALLOCATEDSECTORS / $PENDINGSECTORS"
echo "Read Speed: $SPEED MB/s"

echo
echo


if [ $HOURS -ge 20000 ]; then
echo -e '\033[5;1;37;41mWARNING:\033[0m This drive has over 20,000 hours on it and should not be used as a Primary'
fi

if [ $ERRORCOUNT -gt 0 ]; then
echo -e "\033[5;1;37;41mWARNING:\033[0m The last smart error was $WHENITHAPPEND hours ago, use your judgement to tell if this is relevant (use smartctl -a $DRIVE for more info)"
fi

if [ $REALLOCATEDSECTORS -gt 0 ]; then
echo -e '\033[5;1;37;41mWARNING:\033[0m This drive has some reallocated sectors, this should not be used as a primary and requires judgement if it is to be used for a secondary'
fi

if [ $PENDINGSECTORS -gt 0 ]; then
echo -e '\033[5;1;37;41mWARNING:\033[0m This drive has some pending sectors, this shouldn not be used as a primary and requires judgement if it is to be used for a secondary'
fi

if [ $SPEED -le 25 ]; then
echo -e "\033[5;1;37;41mWARNING:\033[0m This drive is only reading at $SPEED MB/s, which seems slow, is this drive ok and is it on a good motherboard and ide cable?"
fi
\[/code]

Big difference on the results
 
I did have a PERC Raid controller issue, write-back was not set in the bios for this controller (also set adaptive read-ahead whilst making the write change). Now get good performance on the hostmachine (>200MB/sec) but much less performance on a virtualmachine (<30MB/sec). I am using qcow2 as the disk format with virtio drivers.

Here are my benchmarks:-

hostmachine $ dd if=/dev/zero of=/tmp/test oflag=direct bs=128k count=1600
1600+0 records in
1600+0 records out
209715200 bytes (210 MB) copied, 0.925108 s, 227 MB/s
hostmachine $ uname -a
Linux m7 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

And on the Virtual machine I get...

virtualmachine $$ uname -a
Linux vc2pv7 3.2.0-32-virtual #51-Ubuntu SMP Wed Sep 26 21:53:42 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

virtualmachine $ dd if=/dev/zero of=/tmp/test oflag=direct bs=128k count=1600
1600+0 records in
1600+0 records out
209715200 bytes (210 MB) copied, 52.0911 s, 4.0 MB/s
virtualmachine $

Actually I can sometimes get the speed upto around 30MB/s ... but nowhere near the 200MB/s + you get on the hostmachine.

I will not repeat this test having converted the Virtual machine to use LV's.

Dave
 
make sure you do tests with latest 2.2 and newly create qcow2 disk images - 30mb is too low, should be better.
 
VM Proxmox 2.2 with qcow2

hdparm -Tt /dev/vda


/dev/vda:
Timing cached reads: 5914 MB in 2.00 seconds = 2959.22 MB/sec
Timing buffered disk reads: 702 MB in 3.11 seconds = 225.43 MB/sec
[root@dev includes]# hdparm -Tt /dev/vda


/dev/vda:
Timing cached reads: 5714 MB in 2.00 seconds = 2858.70 MB/sec
Timing buffered disk reads: 900 MB in 3.01 seconds = 299.43 MB/sec

dd if=/dev/zero of=/tmp/test oflag=direct bs=128k count=1600
1600+0 records in
1600+0 records out
209715200 bytes (210 MB) copied, 1.05733 s, 198 MB/s


Host node

hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 7816 MB in 2.00 seconds = 3909.91 MB/sec
Timing buffered disk reads: 930 MB in 3.00 seconds = 309.74 MB/sec
root@host-02:~# hdparm -Tt /dev/sda


/dev/sda:
Timing cached reads: 7992 MB in 2.00 seconds = 3997.77 MB/sec
Timing buffered disk reads: 928 MB in 3.01 seconds = 308.41 MB/sec

dd if=/dev/zero of=/tmp/test oflag=direct bs=128k count=1600
1600+0 records in
1600+0 records out
209715200 bytes (210 MB) copied, 0.522191 s, 402 MB/s


Big difference in speed still
 
Okay..now the results when using a LV as the raw disk device (now a block and not a file device)....ITS MUCH FASTER...

virtualmachine $ dd if=/dev/zero of=/tmp/test oflag=direct bs=128k count=1600
1600+0 records in
1600+0 records out
209715200 bytes (210 MB) copied, 0.929538 s, 226 MB/s
virtualmachine $

SO MUCH BETTER PERFORAMCE USING LV's for disk device rather than qcow2 files.

Settings used....

<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source file='/dev/m7/lv_disk'/>
<target dev='vda' bus='virtio'/>


I must confess I'm using Ubuntu 12.04 as the host for these, not proxmox 2.2 ... but this forum seeks the best place to discuss KVM performance issues.

My next task will be to convert Windowsa based machines to using LV disks... does anyone have an equivalent benchmark command to the dd one used above for a windows 2003 machine...so I can post before and after results to this forum.

Thanks Dave
 
...

I must confess I'm using Ubuntu 12.04 as the host for these, not proxmox 2.2 ... but this forum seeks the best place to discuss KVM performance issues.
...

Thanks Dave

KVM on Ubuntu and KVM on Proxmox VE is not the same so posting Ubuntu results here is not that useful ...

Or more detailed, Proxmox VE KVM is much newer and optimized. (Ubuntu 12.04 uses KVM 1.0, we use KVM 1.2)

So switch to Proxmox VE!
 
Thanks for the comments.

Have started to use Crystal Diskmark....thank you for the pointer to use this.

Firstly with my setup as was (Ubuntu 12.04 / KVM 1.0)...

windows2003_CrystalDiskMark_Ubuntu12.04_KVM1.0.png


The ugraded to Ubuntu 12.10 which has KVM 1.2 .... ( I'm happy to use/test Proxmox 2.2 - but that will have to wait for a trip to the datacentre.)
Windows2003_CrystalDiskMark_Ubuntu12.10_KVM1.2.png

Note the randow read performance is much better...but otherwise little change.

Lastly, run the same on a physical server to compare....


gp1_disperfromance.jpg


This physical server has two disks in a Dell poweredge 1950 server.


The botttom line is I feel that the disk perfoamnce issues are now resolved - when I get a chance I'll benchmark on a proxmox 2.2 server ...be interesting if this gives even better results. But having much better disk performance that a physical windows 2003 server is enough for me to be happy for now.

Thanks again for all the help on this one....it has really helped me.

Dave
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!