High I/O wait

[anton00706]

Hello everyone. I have a small problem. Very high iowait.

My configuration:
Supermicro SYS-6026T-URF
2 x Intel Xeon E5620, 8x2400 MHz, 12 Mb
12 x Kingston 8 Гб DDR3-1333 ECC
8 x Hitachi 600 Gb, 15000 rpm, SAS
Adaptec RAID 6805 + ZMM Write-back enabled
Proxmox VE 3.1

I have two KVM machines on this node:
1) Win 2008R2, 2 cpu, 16 Gb RAM, 50 Gb HDD VIRTIO RAW, cache Write-back enabled
2) Win 2008R2, 2 cpu, 32 Gb RAM, 850 Gb HDD VIRTIO RAW, cache Write-back enabled




In what could be the reason?

Regards.

 

[udo]

Hi,
but if I see right you have only 2.x % IO-wait?! This is not very high and should not be an problem.

If your IO-wait are below 5-10% there is no problem.

Udo

 

[macday]

Try to disable the write-back mode in your VM..this usually leads to higher iowait.

cheers
mac

 

[r4pt0x]

caching should be only done by the RAID-controller (thus a BBU is essential!), not by software.
disable HDD-caches at the host and for the guest, then IO-wait and overall HDD-performance should be best.

 

[macday]

caching should be only done by the RAID-controller (thus a BBU is essential!), not by software.
disable HDD-caches at the host and for the guest, then IO-wait and overall HDD-performance should be best.

How do you disable caches on host-level ?

 

[anton00706]

Thank you all for your answers. I'll try to disable the cache on the VM and I will write about the results. Now my server is not a heavy load, but plans add the VM with sql server. So I moved the virtual machine data in raw format.(with Acronis TI), because Wiki says qcow format worse performance. Before it used a qcow2 with write back cache and iowait was less. Wiki says qcow format worse performance. Hmm... does not know what to do

 

[r4pt0x]

on the fly:
hdparm -W0 /dev/sdx

mount-option "sync" and "dirsync" will also bypass the write cache.

Also the guest needs to have it disabled (not only by guest options on the host system!) - on windows guests this can be achieved by enabling quick removal (e.g. on 2008 R2 you also have the option to disable write cache on the drive and in windows).

Otherwise you will stack different caches on top of each other - which is a master recipe for data loss/corruption.
The only cache enabled should be the Raid-Controller cache. The hardware-cache on the drives should also be disabled, as it isn't protected by the BBU in case of an power outage. (3ware has recently added some proprietary caching protocol for this problem)


Also have a look at the article in the wiki:
http://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

Another article about caching and potential data loss:
http://www.thomas-krenn.com/en/wiki/RAID_Controller_and_Hard_Disk_Cache_Settings

 

[anton00706]

on the fly:
hdparm -W0 /dev/sdx

mount-option "sync" and "dirsync" will also bypass the write cache.

Also the guest needs to have it disabled (not only by guest options on the host system!) - on windows guests this can be achieved by enabling quick removal (e.g. on 2008 R2 you also have the option to disable write cache on the drive and in windows).

Otherwise you will stack different caches on top of each other - which is a master recipe for data loss/corruption.
The only cache enabled should be the Raid-Controller cache. The hardware-cache on the drives should also be disabled, as it isn't protected by the BBU in case of an power outage. (3ware has recently added some proprietary caching protocol for this problem)


Also have a look at the article in the wiki:
http://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

Another article about caching and potential data loss:
http://www.thomas-krenn.com/en/wiki/RAID_Controller_and_Hard_Disk_Cache_Settings

Big thx!)