RAID 5 bad write performance

[StefanS]

I'm just running some performance tests on my vserver and the writing speed is pretty slow.

i use bonnie++ for testing and i only reach about 100MB/s (OpenVZ and KVM)
without virtualization i get 200MB/s

is that normal or is it possible to get a better performance?

Board Supermicro H8DME-2 (BIOS v2.58, default) CPU 2x AMD Opteron 2374HE (4x 2,2Ghz) Memory 32GB (8x 4GB 667 ECC) System HDD 1x WD RE3 500GB 5002ABYS RAID-Controller Areca ARC-1220 RAID-HDDs 6x WD RE3 1GB 1002FBYS RAID-Config RAID5 with 5 HDDs + Sparedisk

 

[udo]

Hi,
i'm wondering that you get the same results with openvz and kvm - but better results on the host.
I think with the same parameters you must get a better result on the host, because you have much more memory as the kvm-machine.
Between openvz and kvm you should see differents.
On one node:
kvm

/usr/sbin/bonnie -s 4g:256k -q -f -d test -r 1024                                  
Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
bonnie-test 4G:256k           123625  29 91229  16           342924  42  1032  68
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
bonnie-test,4G:256k,,,123625,29,91229,16,,,342924,42,1031.8,68,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

openvz

/usr/sbin/bonnie -s 4g:256k -q -f -d test -r 1024
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
tine        4G:256k           137286  51 204100  46           2025897  99  2092  33
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++              
tine,4G:256k,,,137286,51,204100,46,,,2025897,99,2092.3,33,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

host

/usr/sbin/bonnie -s 4g:256k -q -f -d test -r 1024
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
proxmox3    4G:256k           177250  58 124284  23           2346176  99 11029 167
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
proxmox3,4G:256k,,,177250,58,124284,23,,,2346176,99,11028.9,167,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

pveperf /var/lib/vz
CPU BOGOMIPS: 10448.36
REGEX/SECOND: 546445
HD SIZE: 444.21 GB (/dev/mapper/pve-data)
BUFFERED READS: 290.73 MB/sec
AVERAGE SEEK TIME: 7.20 ms
FSYNCS/SECOND: 2834.87
DNS EXT: 110.05 ms
DNS INT: 0.61 ms

It's a ARC-1210 Raidcontroller with Raptor 300GB-Sata-disk in raid-10.

Udo

 

[StefanS]

After updating from Proxmox VE 1.4 to 1.5 my results were getting a little better...
but on the kvm machine write performance is still not good (~100MB/s)

host:

bonnie -u root  -s 64g:256k -q -f -d /media/test1 -r 1024
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
euripides  64G:256k           209022  23 95553  10           227598  14 127.5   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 29661  93 +++++ +++ 24388  69 25999  80 +++++ +++ 22301  70
euripides,64G:256k,,,209022,23,95553,10,,,227598,14,127.5,2,16,29661,93,+++++,+++,24388,69,25999,80,+++++,+++,22301,70

when i run it with 4g i get enormous cachingeffects

openvz:

bonnie -u root -s 64g:256k -q -f -d / -r 1024
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
ovz103     64G:256k           204379  23 95247  11           207566  13 119.6   2
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 19705  96 +++++ +++ 21809  58 20384  99 +++++ +++ 19671  61
ovz103,64G:256k,,,204379,23,95247,11,,,207566,13,119.6,2,16,19705,96,+++++,+++,21809,58,20384,99,+++++,+++,19671,61

the same her even though i only gave 4GB of ram

kvm

bonnie -u root -s 4g:256k -q -f -d / -r 1024
Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine   Size:chnk K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
kvm         4G:256k           113963  22 90358  16           211005  12 861.7  11
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
kvm,4G:256k,,,113963,22,90358,16,,,211005,12,861.7,11,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

as you can see the write speed ist 113963 K/s


how much memory do you have?
i am wondering why your reading speed is 2346176 K/s (~2300 MB/s) directly on your host...


pveperf /media/
CPU BOGOMIPS: 35376.76
REGEX/SECOND: 730930
HD SIZE: 3725.17 GB (/dev/sdb1)
BUFFERED READS: 304.04 MB/sec
AVERAGE SEEK TIME: 12.65 ms
FSYNCS/SECOND: 1244.93
DNS EXT: 35.60 ms
DNS INT: 15.75 ms

 

[udo]

...

how much memory do you have?
i am wondering why your reading speed is 2346176 K/s (~2300 MB/s) directly on your host...

...

Hi,
on this host i have 8GB ram - so i need minimum 16GB testfile for bonnie. With my 4GB i test the RAM.
But the kvm-host has only 1.5 G Ram - so the value are normal.

On the host with 16GB:

proxmox3,16G:256k,,,142627,46,101281,26,,,286188,23,218.5,6,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

With raid-5 is the write-performance only medium - therefor many people in this forum recomend raid 10.

Udo

 

[StefanS]

what i don't really understand is, that my writing speed on a kvm machine is about 50% slower compared to the tests i ran directly on the host and on a openvz machine...