KVM performance on different operating systems

@docnet: Please can you use fio to do the benchmarks. That way we can try to reproduce the behavior here.
 
dietmar said:
NO, barrier=1 is safer (and much slower).
Click to expand...

I've remounted /var/lib/vz with the options like in CentOS except 'seclabel'.
Code: 
kernel: EXT3-fs (dm-5): error: unrecognized mount option "seclabel" or missing value
Code: 
/dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,barrier=1,data=ordered 0 0
The result did not change.:(
 
here my fio result, 6 disk15k raid10, dell perc5 (old) card with battery, writeback enable (512MB cache)
using defaults /dev/pve/data /var/lib/vz ext3 defaults 0 1

fio --filename=/var/lib/vz/testbench --direct=1 --rw=randread --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
random read: 5000iopss
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randwrite --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
random write : 4500iops
 
using mount with option 'barrier=1':
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randread --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
file1: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [75984K/0K /s] [19K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=41508
  read : io=1024MB, bw=84692KB/s, iops=21173, runt= 12381msec
    slat (usec): min=9, max=993, avg=20.44, stdev= 7.16
    clat (usec): min=2, max=309558, avg=4698.42, stdev=14310.68
    bw (KB/s) : min=73096, max=97904, per=100.49%, avg=85104.33, stdev=6532.32
  cpu          : usr=9.85%, sys=45.53%, ctx=91895, majf=0, minf=123
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=262144/0, short=0/0
     lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=33.34%, 100=28.87%
     lat (usec): 250=6.88%, 500=0.56%, 750=0.35%, 1000=0.34%
     lat (msec): 2=2.26%, 4=5.47%, 10=9.98%, 20=5.63%, 50=4.36%
     lat (msec): 100=1.48%, 250=0.48%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=1024MB, aggrb=84692KB/s, minb=86724KB/s, maxb=86724KB/s, mint=12381msec, maxt=12381msec

Disk stats (read/write):
  dm-5: ios=257997/288, merge=0/0, ticks=1198622/11128, in_queue=1212014, util=99.28%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randwrite --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1): [w] [97.1% done] [0K/38590K /s] [0/9421 iops] [eta 00m:01s]
file1: (groupid=0, jobs=1): err= 0: pid=43979
  write: io=1024MB, bw=31752KB/s, iops=7937, runt= 33024msec
    slat (usec): min=8, max=132596, avg=19.72, stdev=259.21
    clat (usec): min=27, max=712650, avg=12573.27, stdev=23886.56
    bw (KB/s) : min=22384, max=121288, per=99.90%, avg=31720.45, stdev=11700.06
  cpu          : usr=3.10%, sys=15.76%, ctx=19914, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=0/262144, short=0/0
     lat (usec): 50=0.03%, 100=0.08%, 250=0.27%, 500=0.47%, 750=0.86%
     lat (usec): 1000=0.64%
     lat (msec): 2=3.39%, 4=16.04%, 10=40.95%, 20=27.62%, 50=6.85%
     lat (msec): 100=1.13%, 250=1.47%, 500=0.19%, 750=0.01%

Run status group 0 (all jobs):
  WRITE: io=1024MB, aggrb=31751KB/s, minb=32513KB/s, maxb=32513KB/s, mint=33024msec, maxt=33024msec

Disk stats (read/write):
  dm-5: ios=50/259335, merge=0/0, ticks=6636/3185608, in_queue=3196368, util=99.80%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
using mount with option 'barrier=0':
Code: 
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randread --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [28434K/0K /s] [6942/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2832
  read : io=1024MB, bw=28544KB/s, iops=7136, runt= 36735msec
    slat (usec): min=8, max=11840, avg=22.56, stdev=130.98
    clat (usec): min=109, max=451335, avg=13926.42, stdev=16772.07
    bw (KB/s) : min= 1115, max=30344, per=100.30%, avg=28628.53, stdev=3913.98
  cpu          : usr=4.46%, sys=12.58%, ctx=197408, majf=0, minf=136
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=262144/0, short=0/0
     lat (usec): 250=0.02%, 500=0.02%, 750=0.05%, 1000=0.34%
     lat (msec): 2=6.31%, 4=17.07%, 10=33.18%, 20=22.35%, 50=16.86%
     lat (msec): 100=3.27%, 250=0.52%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=1024MB, aggrb=28544KB/s, minb=29229KB/s, maxb=29229KB/s, mint=36735msec, maxt=36735msec

Disk stats (read/write):
  dm-5: ios=262251/3, merge=0/0, ticks=3574939/0, in_queue=3576542, util=99.81%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randwrite --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0K/44139K /s] [0/11K iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2959
  write: io=1024MB, bw=31493KB/s, iops=7873, runt= 33296msec
    slat (usec): min=8, max=1016, avg=19.01, stdev=10.44
    clat (usec): min=35, max=590981, avg=12677.59, stdev=24785.25
    bw (KB/s) : min=    0, max=125197, per=40.26%, avg=12677.61, stdev=17262.53
  cpu          : usr=3.39%, sys=14.89%, ctx=17741, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=0/262144, short=0/0
     lat (usec): 50=0.06%, 100=0.16%, 250=0.68%, 500=0.82%, 750=0.67%
     lat (usec): 1000=0.46%
     lat (msec): 2=6.02%, 4=13.81%, 10=40.41%, 20=26.69%, 50=7.48%
     lat (msec): 100=1.09%, 250=1.44%, 500=0.21%, 750=0.01%

Run status group 0 (all jobs):
  WRITE: io=1024MB, aggrb=31492KB/s, minb=32248KB/s, maxb=32248KB/s, mint=33296msec, maxt=33296msec

Disk stats (read/write):
  dm-5: ios=1/260135, merge=0/0, ticks=47/3210901, in_queue=3214280, util=99.79%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=read --bs=4M --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=read, bs=4M-4M/4M-4M, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1)
file1: (groupid=0, jobs=1): err= 0: pid=4781
  read : io=1024MB, bw=794978KB/s, iops=194, runt=  1319msec
    slat (usec): min=601, max=27543, avg=4953.98, stdev=4645.99
    clat (msec): min=49, max=617, avg=425.22, stdev=166.69
    bw (KB/s) : min=16384, max=713758, per=45.92%, avg=365071.00, stdev=493117.88
  cpu          : usr=0.00%, sys=33.31%, ctx=119, majf=0, minf=102426
  IO depths    : 1=0.4%, 2=0.8%, 4=1.6%, 8=3.1%, 16=6.2%, 32=12.5%, >=64=75.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.6%
     issued r/w: total=256/0, short=0/0

     lat (msec): 50=0.39%, 100=4.69%, 250=15.23%, 500=32.03%, 750=47.66%

Run status group 0 (all jobs):
   READ: io=1024MB, aggrb=794978KB/s, minb=814057KB/s, maxb=814057KB/s, mint=1319msec, maxt=1319msec

Disk stats (read/write):
  dm-5: ios=5789/0, merge=0/0, ticks=223223/0, in_queue=228474, util=91.91%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=write --bs=4M --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=write, bs=4M-4M/4M-4M, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1)
file1: (groupid=0, jobs=1): err= 0: pid=4809
  write: io=1024MB, bw=1463MB/s, iops=365, runt=   700msec
    slat (usec): min=506, max=22554, avg=2446.99, stdev=4010.51
    clat (msec): min=71, max=519, avg=250.95, stdev=131.57
    bw (KB/s) : min=1035953, max=1035953, per=69.16%, avg=1035953.00, stdev= 0.00
  cpu          : usr=0.00%, sys=22.75%, ctx=125, majf=0, minf=25
  IO depths    : 1=0.4%, 2=0.8%, 4=1.6%, 8=3.1%, 16=6.2%, 32=12.5%, >=64=75.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.6%
     issued r/w: total=0/256, short=0/0

     lat (msec): 100=3.52%, 250=51.56%, 500=41.41%, 750=3.52%

Run status group 0 (all jobs):
  WRITE: io=1024MB, aggrb=1463MB/s, minb=1498MB/s, maxb=1498MB/s, mint=700msec, maxt=700msec

Disk stats (read/write):
  dm-5: ios=0/6181, merge=0/0, ticks=0/124102, in_queue=129468, util=24.44%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
 
So, everything seem to be really fast, iops or bandwith on the hardware side.

So, the slowdown must be come from qemu-kvm or windows guest version.
Maybe they is some difference in qemu-kvm version ?
can you provide #kvm -h or qemu -h for each host os ?
 
also, maybe could you try with the new virtio-scsi controller ? (choose virtio-scsi controller in vm options, and set disk to scsi)

I never test it under windows but I think that drivers are available in last fedora virtio win iso.
 
I think that it is not in the storage system.
This is the diagram of CPU load on Proxmox during the execution of my program:
Proxmox.png
This is the diagram of CPU load on CentOS:
CentOS.png
This is the result of program ibbackup.exe v6.5.1.887 from Yaffil SQL Server v.1.3.0.887.
 
Last edited:
docent said:
In Proxmox /dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,barrier=0,data=ordered 0 0
In CentOS /dev/mapper/pve-data /mnt/data ext3 rw,seclabel,relatime,errors=continue,barrier=1,data=ordered 0 0

I've tested with one VCPU, results are the same.
Click to expand...
Hi,
I don't mean VCPU - I mean only 1 real CPU. Because I have also issues with an 4-CPU system, which not reproducable on 1-CPU-systems (in this cases with networkcards/iScsi - I suspect the Supermicroboard, but perhaps it's something else.).

Udo
 
spirit said:
So, the slowdown must be come from qemu-kvm or windows guest version.
Maybe they is some difference in qemu-kvm version ?
can you provide #kvm -h or qemu -h for each host os ?
Click to expand...

I'm trying to find difference between Proxmox and CentOS because everything else is the same.
In CentOS:
Code: 
# /usr/libexec/qemu-kvm --version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hg-sr-v1 2.6.32-279.5.2.el6.x86_64 #1 SMP Fri Aug 24 01:07:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

In Proxmox:
Code: 
# kvm --version
QEMU emulator version 1.0.50 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hq-sr-v1 2.6.32-11-pve #1 SMP Wed Apr 11 07:17:05 CEST 2012 x86_64 GNU/Linux
 
docent said:
I'm trying to find difference between Proxmox and CentOS because everything else is the same.
In CentOS:
Code: 
# /usr/libexec/qemu-kvm --version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hg-sr-v1 2.6.32-279.5.2.el6.x86_64 #1 SMP Fri Aug 24 01:07:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

In Proxmox:
Code: 
# kvm --version
QEMU emulator version 1.0.50 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hq-sr-v1 2.6.32-11-pve #1 SMP Wed Apr 11 07:17:05 CEST 2012 x86_64 GNU/Linux
Click to expand...

- centos/redhat use old kvm 0.12 (with a lot of patches)

but for proxmox, you shouldn't have qemu 1.0.50 , it should be 1.1. (or 1.2 in pvetest).
Do you have updated your proxmox installation ????
 
spirit said:
- centos/redhat use old kvm 0.12 (with a lot of patches)

but for proxmox, you shouldn't have qemu 1.0.50 , it should be 1.1. (or 1.2 in pvetest).
Do you have updated your proxmox installation ????
Click to expand...
apt-get upgrade didn't work. I had to run aptitude full-upgrade.
Now I have the next:
Code: 
# kvm --version
QEMU emulator version 1.1.1 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hq-sr-v1 2.6.32-14-pve #1 SMP Tue Aug 21 08:24:37 CEST 2012 x86_64 GNU/Linux

But I still get exactly the same result in VM. :(
 
spirit said:
maybe can you retry the proxmox qemu command line, and use
smp 2,sockets=2,cores=1,threads=1
Click to expand...
For the tests I use VM with only one VCPU.

Are there any benchmarks for various virtualization subsystems that I could run?
I've already tested the storage subsystem, performance is the same in both operation systems.
 
Last edited:
do you done benchmarks inside your windows vm?
like iometer for disk, or cpumark for cpu ?

One other thing you can try:
-try to install kvm package from ubuntu to compare.
-try to use debian 3.2 kernel from squeeze backport (proxmox kernel have transparent hugepages desactivated because of an openvz )
 
spirit said:
do you done benchmarks inside your windows vm?
like iometer for disk, or cpumark for cpu ?

One other thing you can try:
-try to install kvm package from ubuntu to compare.
-try to use debian 3.2 kernel from squeeze backport (proxmox kernel have transparent hugepages desactivated because of an openvz )
Click to expand...

I found free benchmark and run on VM on both OS.
As you can see, only file i/o is better in Proxmox.
I've tested ubuntu 12.04 http://forum.proxmox.com/threads/11...different-operating-systems?p=61720#post61720
It's performance is closer to CentOS.
I know, that there is the kernel in Proxmox which supports only KVM w/o OpenVZ. How can I install it?
 

Attachments

Last edited:
ok, so we are sure it's not disk related.

maybe cpu or ram.
- can you try to use "cpu64-rhel6" as cpu model ? (to be equal as centos)
- for memory, it could be related to transparent hugepage, so try the debian kernel 3.2 from backports

add

"deb http://backports.debian.org/debian-backports squeeze-backports main"

to /etc/apt/sources.list

then
#apt-get update
# apt-get install -t squeeze-backports linux-image-3.2.0-0.bpo.3-amd64
 
You must log in or register to reply here.
Top Bottom