VERY slow disk, and CPU throttling errors

[tytanick]

hi guys, i have very big problem with my proxmox.
J have two 7200rpm Red WD 2x 2TB, and my system has all the time 50-90% budy disk /dev/sda (i see this in atop)
Disks are connected via: INTEL RAID SASWT4I

Tell me, is my FSYNCS/SECOND: 6.48 isnt too low ?
What can i do ?

My system:
Linux proxmox 2.6.32-23-pve #1 SMP Tue Aug 6 07:04:06 CEST 2013 x86_64 GNU/Linux

pveperf:
CPU BOGOMIPS: 55998.72
REGEX/SECOND: 2636915
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 76.33 MB/sec
AVERAGE SEEK TIME: 20.17 ms
FSYNCS/SECOND: 6.48
DNS EXT: 77.37 ms
DNS INT: 0.60 ms (vipro.eu)

hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 28080 MB in 2.00 seconds = 14055.54 MB/sec
Timing buffered disk reads: 418 MB in 3.01 seconds = 138.89 MB/sec

iostat -d -x -m
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 9.21 65.29 35.70 21.31 0.76 0.46 43.78 1.21 21.19 5.65 47.24 8.38 47.79
dm-0 0.00 0.00 5.09 4.16 0.03 0.02 9.62 0.95 102.37 5.36 221.06 3.45 3.19
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 0.72 0.72 0.00 0.46 0.00
dm-2 0.00 0.00 36.54 82.44 0.72 0.44 20.01 26.97 226.66 5.51 324.69 3.98 47.37



Also dmesg show me many problems with cpu temperature ?
CPU2: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU0: Core temperature above threshold, cpu clock throttled (total events = 5090)
CPU4: Core temperature above threshold, cpu clock throttled (total events = 5090)
CPU6: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU1: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU5: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU7: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU3: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU0: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU4: Package temperature above threshold, cpu clock throttled (total events = 8053)
CPU0: Core temperature/speed normal
CPU5: Package temperature/speed normal
CPU4: Core temperature/speed normal
CPU1: Package temperature/speed normal
CPU6: Package temperature/speed normal
CPU3: Package temperature/speed normal
CPU7: Package temperature/speed normal
CPU0: Package temperature/speed normal
CPU4: Package temperature/speed normal
CPU2: Package temperature/speed normal

 

[m.ardito]

Has your raid controller a BBU, and it's active & charged?
That could be a reason... most BBU backed raid controller switch to writethrough when BBU is not good or charged.

easy description: http://www.intel.com/support/motherboards/server/sb/CS-023687.htm

Marco

 

[tytanick]

Has your raid controller a BBU, and it's active & charged?
That could be a reason... most BBU backed raid controller switch to writethrough when BBU is not good or charged.

easy description: http://www.intel.com/support/motherboards/server/sb/CS-023687.htm

Marco

I checked and i dont even have BBU option in raid, this raid doesnt support BBU.
So again, accually hdparm -tT /dev/sda is pretty good results, but when i am starting VM, i have 95% busy /dev/sda and only 7MB/s read, so what the heck ?

P.S - i even checked performance WITHOUT RAID, so with direct disk it was exactly the same !
I even changed motherboard do different one, and still the same.
So it must be something in software, kernel or VM options ?
I change form no cache IDE in VM's to WriteBack, no change, still the same issue
Check this out, i have even 2MB/s and 90% busy
And i dont have CPU throttling errors anymore /9because i changed motherboard and cpu and cpu cooling)

 

[e100]

Those WD RED drives do not have rotational vibration sensors, the WD RAID Edition drives do.

I once built a server on a budget. Sitting on my desk it ran great, in the server rack disk IO was horrible. The difference was vibrations from all the other servers. Swapped the disk with one that did have rotational vibration sensors and the problem was resolved.

Not sure if that is your problem or not but it could be. Sometimes the hardware and the environment it is used in does matter.

Maybe your yelling at the slow server is the cause ;-)
http://classic.slashdot.org/story/09/01/02/0626201

 

[e100]

Oh, one other thing.

What does pveperf report when all the VMs are stopped?

Maybe the VMs are doing lots of IO thus making pveperf report slower fsyncs than what the hardware is capable of.

 

[tytanick]

Another results when i have 4 VM powered on.
In about 8-10 hours i will make the same tests on my proxmox with powered OFF all VMs.
As i see it, system alone is working pretty good, but when VMs are getting powered on, whole system slows down, perhaps there is something wrong with one VM.
Meaby QCOW2 isnt good and i should try raw ? (one VM is windows 2003 or 2008 with SQL on it)



TEST WITH 4 VMs POWERED ON:

root@proxmox:~# time (dd if=/dev/zero of=/test.dd bs=4k count=400000; sync)
400000+0 records in
400000+0 records out
1638400000 bytes (1.6 GB) copied, 2.85297 s, 574 MB/s
real 0m33.233s
user 0m0.047s
sys 0m1.843s

root@proxmox:~# dd if=/dev/zero of=/blah.dd bs=4k count=400000 conv=fdatasync
400000+0 records in
400000+0 records out
1638400000 bytes (1.6 GB) copied, 27.647 s, 59.3 MB/s

root@proxmox:~# hdparm -tT /dev/sda
/dev/sda:
Timing cached reads: 14570 MB in 2.00 seconds = 7291.15 MB/sec
Timing buffered disk reads: 176 MB in 3.00 seconds = 58.61 MB/sec

root@proxmox:~# pveperf
CPU BOGOMIPS: 54277.44
REGEX/SECOND: 1452561
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 34.26 MB/sec
AVERAGE SEEK TIME: 24.43 ms
FSYNCS/SECOND: 12.57
DNS EXT: 56.02 ms
DNS INT: 1.22 ms (vipro.eu)

 

[tytanick]

ALL powered OFF:

root@proxmox:~# pveperf
CPU BOGOMIPS: 54277.44
REGEX/SECOND: 1760548
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 69.04 MB/sec
AVERAGE SEEK TIME: 25.58 ms
FSYNCS/SECOND: 14.30
DNS EXT: 57.26 ms
DNS INT: 0.73 ms (vipro.eu)

it must be something elsee

 

[mmenaz]

Are you using ext3 or ext4 (or something different)?

 

[e100]

Many RAID setups disable the cache that is built into the disks, this can cause horrible performance.
If the disk write cache is on you risk loosing data if you have a power outage.

Maybe there is a setting in your RAID that you can change to enable the disk write cache.

 

[tytanick]

I cant find anything in raid configuration utulity from bios, but i will once again try to connect disk directly to motherboard without raid controler and check again performance.

If raid somehow disablec cache biuild in to disk, and i will connect this disk directly to motherboard without raid controlerr, will this cache will be still disabled and i have to i dont know, recreate partitions ? (this is stupid ...)
Or it can only disable disk cahce only if the drive is in raid ?

About filesystem: its ext3

# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pve/root / ext3 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext3 defaults 0 1
UUID=fc7d672f-71e6-461a-9654-ac9fab46e15f /boot ext3 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

 

[tytanick]

tell me is ext3 that much less performance than ext4 ?

[ 7.768907] EXT4-fs (dm-2): mounting ext3 file system using the ext4 subsystem
[ 7.806558] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[ 7.811829] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[ 7.833954] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[ 7.910586] EXT4-fs (sdb1): mounted filesystem with ordered data mode. Opts: barrier=1,nodelalloc,data=ordered

 

[tytanick]

i changed motherboard, took off hardware RAID, there is no raid now.
even changed i7 4 gen to i7 3 gen - still the same issue.
a lot of IO delay.