KVM performance on different operating systems

@docnet: Please can you use fio to do the benchmarks. That way we can try to reproduce the behavior here.
 
dietmar said:
NO, barrier=1 is safer (and much slower).
Click to expand...

I've remounted /var/lib/vz with the options like in CentOS except 'seclabel'.
Code: 
kernel: EXT3-fs (dm-5): error: unrecognized mount option "seclabel" or missing value
Code: 
/dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,barrier=1,data=ordered 0 0
The result did not change.:(
 
here my fio result, 6 disk15k raid10, dell perc5 (old) card with battery, writeback enable (512MB cache)
using defaults /dev/pve/data /var/lib/vz ext3 defaults 0 1

fio --filename=/var/lib/vz/testbench --direct=1 --rw=randread --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
random read: 5000iopss
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randwrite --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
random write : 4500iops
 
using mount with option 'barrier=1':
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randread --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
file1: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [r] [100.0% done] [75984K/0K /s] [19K/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=41508
  read : io=1024MB, bw=84692KB/s, iops=21173, runt= 12381msec
    slat (usec): min=9, max=993, avg=20.44, stdev= 7.16
    clat (usec): min=2, max=309558, avg=4698.42, stdev=14310.68
    bw (KB/s) : min=73096, max=97904, per=100.49%, avg=85104.33, stdev=6532.32
  cpu          : usr=9.85%, sys=45.53%, ctx=91895, majf=0, minf=123
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=262144/0, short=0/0
     lat (usec): 4=0.01%, 10=0.01%, 20=0.01%, 50=33.34%, 100=28.87%
     lat (usec): 250=6.88%, 500=0.56%, 750=0.35%, 1000=0.34%
     lat (msec): 2=2.26%, 4=5.47%, 10=9.98%, 20=5.63%, 50=4.36%
     lat (msec): 100=1.48%, 250=0.48%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=1024MB, aggrb=84692KB/s, minb=86724KB/s, maxb=86724KB/s, mint=12381msec, maxt=12381msec

Disk stats (read/write):
  dm-5: ios=257997/288, merge=0/0, ticks=1198622/11128, in_queue=1212014, util=99.28%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randwrite --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1): [w] [97.1% done] [0K/38590K /s] [0/9421 iops] [eta 00m:01s]
file1: (groupid=0, jobs=1): err= 0: pid=43979
  write: io=1024MB, bw=31752KB/s, iops=7937, runt= 33024msec
    slat (usec): min=8, max=132596, avg=19.72, stdev=259.21
    clat (usec): min=27, max=712650, avg=12573.27, stdev=23886.56
    bw (KB/s) : min=22384, max=121288, per=99.90%, avg=31720.45, stdev=11700.06
  cpu          : usr=3.10%, sys=15.76%, ctx=19914, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=0/262144, short=0/0
     lat (usec): 50=0.03%, 100=0.08%, 250=0.27%, 500=0.47%, 750=0.86%
     lat (usec): 1000=0.64%
     lat (msec): 2=3.39%, 4=16.04%, 10=40.95%, 20=27.62%, 50=6.85%
     lat (msec): 100=1.13%, 250=1.47%, 500=0.19%, 750=0.01%

Run status group 0 (all jobs):
  WRITE: io=1024MB, aggrb=31751KB/s, minb=32513KB/s, maxb=32513KB/s, mint=33024msec, maxt=33024msec

Disk stats (read/write):
  dm-5: ios=50/259335, merge=0/0, ticks=6636/3185608, in_queue=3196368, util=99.80%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
using mount with option 'barrier=0':
Code: 
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randread --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
file1: (g=0): rw=randread, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1): [r] [100.0% done] [28434K/0K /s] [6942/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2832
  read : io=1024MB, bw=28544KB/s, iops=7136, runt= 36735msec
    slat (usec): min=8, max=11840, avg=22.56, stdev=130.98
    clat (usec): min=109, max=451335, avg=13926.42, stdev=16772.07
    bw (KB/s) : min= 1115, max=30344, per=100.30%, avg=28628.53, stdev=3913.98
  cpu          : usr=4.46%, sys=12.58%, ctx=197408, majf=0, minf=136
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=262144/0, short=0/0
     lat (usec): 250=0.02%, 500=0.02%, 750=0.05%, 1000=0.34%
     lat (msec): 2=6.31%, 4=17.07%, 10=33.18%, 20=22.35%, 50=16.86%
     lat (msec): 100=3.27%, 250=0.52%, 500=0.01%

Run status group 0 (all jobs):
   READ: io=1024MB, aggrb=28544KB/s, minb=29229KB/s, maxb=29229KB/s, mint=36735msec, maxt=36735msec

Disk stats (read/write):
  dm-5: ios=262251/3, merge=0/0, ticks=3574939/0, in_queue=3576542, util=99.81%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=randwrite --bs=4k --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=randwrite, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1): [w] [100.0% done] [0K/44139K /s] [0/11K iops] [eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=2959
  write: io=1024MB, bw=31493KB/s, iops=7873, runt= 33296msec
    slat (usec): min=8, max=1016, avg=19.01, stdev=10.44
    clat (usec): min=35, max=590981, avg=12677.59, stdev=24785.25
    bw (KB/s) : min=    0, max=125197, per=40.26%, avg=12677.61, stdev=17262.53
  cpu          : usr=3.39%, sys=14.89%, ctx=17741, majf=0, minf=35
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued r/w: total=0/262144, short=0/0
     lat (usec): 50=0.06%, 100=0.16%, 250=0.68%, 500=0.82%, 750=0.67%
     lat (usec): 1000=0.46%
     lat (msec): 2=6.02%, 4=13.81%, 10=40.41%, 20=26.69%, 50=7.48%
     lat (msec): 100=1.09%, 250=1.44%, 500=0.21%, 750=0.01%

Run status group 0 (all jobs):
  WRITE: io=1024MB, aggrb=31492KB/s, minb=32248KB/s, maxb=32248KB/s, mint=33296msec, maxt=33296msec

Disk stats (read/write):
  dm-5: ios=1/260135, merge=0/0, ticks=47/3210901, in_queue=3214280, util=99.79%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=read --bs=4M --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=read, bs=4M-4M/4M-4M, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1)
file1: (groupid=0, jobs=1): err= 0: pid=4781
  read : io=1024MB, bw=794978KB/s, iops=194, runt=  1319msec
    slat (usec): min=601, max=27543, avg=4953.98, stdev=4645.99
    clat (msec): min=49, max=617, avg=425.22, stdev=166.69
    bw (KB/s) : min=16384, max=713758, per=45.92%, avg=365071.00, stdev=493117.88
  cpu          : usr=0.00%, sys=33.31%, ctx=119, majf=0, minf=102426
  IO depths    : 1=0.4%, 2=0.8%, 4=1.6%, 8=3.1%, 16=6.2%, 32=12.5%, >=64=75.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.6%
     issued r/w: total=256/0, short=0/0

     lat (msec): 50=0.39%, 100=4.69%, 250=15.23%, 500=32.03%, 750=47.66%

Run status group 0 (all jobs):
   READ: io=1024MB, aggrb=794978KB/s, minb=814057KB/s, maxb=814057KB/s, mint=1319msec, maxt=1319msec

Disk stats (read/write):
  dm-5: ios=5789/0, merge=0/0, ticks=223223/0, in_queue=228474, util=91.91%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
fio --filename=/var/lib/vz/testbench --direct=1 --rw=write --bs=4M --size=1G --iodepth=100 --runtime=120 --group_reporting --name=file1 --ioengine=libaio
Code: 
file1: (g=0): rw=write, bs=4M-4M/4M-4M, ioengine=libaio, iodepth=100
Starting 1 process
Jobs: 1 (f=1)
file1: (groupid=0, jobs=1): err= 0: pid=4809
  write: io=1024MB, bw=1463MB/s, iops=365, runt=   700msec
    slat (usec): min=506, max=22554, avg=2446.99, stdev=4010.51
    clat (msec): min=71, max=519, avg=250.95, stdev=131.57
    bw (KB/s) : min=1035953, max=1035953, per=69.16%, avg=1035953.00, stdev= 0.00
  cpu          : usr=0.00%, sys=22.75%, ctx=125, majf=0, minf=25
  IO depths    : 1=0.4%, 2=0.8%, 4=1.6%, 8=3.1%, 16=6.2%, 32=12.5%, >=64=75.4%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=99.4%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.6%
     issued r/w: total=0/256, short=0/0

     lat (msec): 100=3.52%, 250=51.56%, 500=41.41%, 750=3.52%

Run status group 0 (all jobs):
  WRITE: io=1024MB, aggrb=1463MB/s, minb=1498MB/s, maxb=1498MB/s, mint=700msec, maxt=700msec

Disk stats (read/write):
  dm-5: ios=0/6181, merge=0/0, ticks=0/124102, in_queue=129468, util=24.44%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    sda: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan%
 
So, everything seem to be really fast, iops or bandwith on the hardware side.

So, the slowdown must be come from qemu-kvm or windows guest version.
Maybe they is some difference in qemu-kvm version ?
can you provide #kvm -h or qemu -h for each host os ?
 
also, maybe could you try with the new virtio-scsi controller ? (choose virtio-scsi controller in vm options, and set disk to scsi)

I never test it under windows but I think that drivers are available in last fedora virtio win iso.
 
I think that it is not in the storage system.
This is the diagram of CPU load on Proxmox during the execution of my program:
Proxmox.png
This is the diagram of CPU load on CentOS:
CentOS.png
This is the result of program ibbackup.exe v6.5.1.887 from Yaffil SQL Server v.1.3.0.887.
 
Last edited:
docent said:
In Proxmox /dev/mapper/pve-data /var/lib/vz ext3 rw,relatime,errors=continue,barrier=0,data=ordered 0 0
In CentOS /dev/mapper/pve-data /mnt/data ext3 rw,seclabel,relatime,errors=continue,barrier=1,data=ordered 0 0

I've tested with one VCPU, results are the same.
Click to expand...
Hi,
I don't mean VCPU - I mean only 1 real CPU. Because I have also issues with an 4-CPU system, which not reproducable on 1-CPU-systems (in this cases with networkcards/iScsi - I suspect the Supermicroboard, but perhaps it's something else.).

Udo
 
spirit said:
So, the slowdown must be come from qemu-kvm or windows guest version.
Maybe they is some difference in qemu-kvm version ?
can you provide #kvm -h or qemu -h for each host os ?
Click to expand...

I'm trying to find difference between Proxmox and CentOS because everything else is the same.
In CentOS:
Code: 
# /usr/libexec/qemu-kvm --version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hg-sr-v1 2.6.32-279.5.2.el6.x86_64 #1 SMP Fri Aug 24 01:07:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

In Proxmox:
Code: 
# kvm --version
QEMU emulator version 1.0.50 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hq-sr-v1 2.6.32-11-pve #1 SMP Wed Apr 11 07:17:05 CEST 2012 x86_64 GNU/Linux
 
docent said:
I'm trying to find difference between Proxmox and CentOS because everything else is the same.
In CentOS:
Code: 
# /usr/libexec/qemu-kvm --version
QEMU PC emulator version 0.12.1 (qemu-kvm-0.12.1.2), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hg-sr-v1 2.6.32-279.5.2.el6.x86_64 #1 SMP Fri Aug 24 01:07:11 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

In Proxmox:
Code: 
# kvm --version
QEMU emulator version 1.0.50 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hq-sr-v1 2.6.32-11-pve #1 SMP Wed Apr 11 07:17:05 CEST 2012 x86_64 GNU/Linux
Click to expand...

- centos/redhat use old kvm 0.12 (with a lot of patches)

but for proxmox, you shouldn't have qemu 1.0.50 , it should be 1.1. (or 1.2 in pvetest).
Do you have updated your proxmox installation ????
 
spirit said:
- centos/redhat use old kvm 0.12 (with a lot of patches)

but for proxmox, you shouldn't have qemu 1.0.50 , it should be 1.1. (or 1.2 in pvetest).
Do you have updated your proxmox installation ????
Click to expand...
apt-get upgrade didn't work. I had to run aptitude full-upgrade.
Now I have the next:
Code: 
# kvm --version
QEMU emulator version 1.1.1 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard
# uname -a
Linux hq-sr-v1 2.6.32-14-pve #1 SMP Tue Aug 21 08:24:37 CEST 2012 x86_64 GNU/Linux

But I still get exactly the same result in VM. :(
 
spirit said:
maybe can you retry the proxmox qemu command line, and use
smp 2,sockets=2,cores=1,threads=1
Click to expand...
For the tests I use VM with only one VCPU.

Are there any benchmarks for various virtualization subsystems that I could run?
I've already tested the storage subsystem, performance is the same in both operation systems.
 
Last edited:
do you done benchmarks inside your windows vm?
like iometer for disk, or cpumark for cpu ?

One other thing you can try:
-try to install kvm package from ubuntu to compare.
-try to use debian 3.2 kernel from squeeze backport (proxmox kernel have transparent hugepages desactivated because of an openvz )
 
spirit said:
do you done benchmarks inside your windows vm?
like iometer for disk, or cpumark for cpu ?

One other thing you can try:
-try to install kvm package from ubuntu to compare.
-try to use debian 3.2 kernel from squeeze backport (proxmox kernel have transparent hugepages desactivated because of an openvz )
Click to expand...

I found free benchmark and run on VM on both OS.
As you can see, only file i/o is better in Proxmox.
I've tested ubuntu 12.04 http://forum.proxmox.com/threads/11...different-operating-systems?p=61720#post61720
It's performance is closer to CentOS.
I know, that there is the kernel in Proxmox which supports only KVM w/o OpenVZ. How can I install it?
 

Attachments

  • Proxmox-CentOS.png
    Proxmox-CentOS.png
    75.8 KB · Views: 28
  • Overal Score Proxmox.txt
    3.5 KB · Views: 25
  • Overall Score CentOS.txt
    3.5 KB · Views: 12
Last edited:
ok, so we are sure it's not disk related.

maybe cpu or ram.
- can you try to use "cpu64-rhel6" as cpu model ? (to be equal as centos)
- for memory, it could be related to transparent hugepage, so try the debian kernel 3.2 from backports

add

"deb http://backports.debian.org/debian-backports squeeze-backports main"

to /etc/apt/sources.list

then
#apt-get update
# apt-get install -t squeeze-backports linux-image-3.2.0-0.bpo.3-amd64
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back