Proxmox 2.3 High IOwait on 2 cpu's out of 12

[ndew]

Hi,

I'm trying to track down where the IOwait is coming from, 2 of the 12 cpu's in the system are almost always higher then 80% IOwait the other cpu's spike occasionally as i would expect, load on the server is hovering around 4-8 and things are responsive, however there must be a reason the IOwait is high on just those two cpus all the time. They are cpu 3 and 10 so they are not near each other

Cpu0 : 9.0%us, 2.7%sy, 57.0%ni, 28.5%id, 2.3%wa, 0.0%hi, 0.5%si, 0.0%st
Cpu1 : 36.5%us, 1.8%sy, 0.0%ni, 61.3%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 20.2%us, 0.4%sy, 0.9%ni, 77.1%id, 1.3%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 3.2%us, 2.7%sy, 36.5%ni, 0.0%id, 57.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 4.9%us, 1.3%sy, 0.0%ni, 93.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 6.7%us, 1.3%sy, 0.0%ni, 91.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 9.5%us, 2.7%sy, 0.0%ni, 87.4%id, 0.0%wa, 0.0%hi, 0.5%si, 0.0%st
Cpu7 : 16.1%us, 1.8%sy, 0.0%ni, 82.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu8 : 0.9%us, 1.8%sy, 0.0%ni, 96.4%id, 0.9%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu9 : 2.2%us, 1.3%sy, 0.0%ni, 96.4%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu10 : 5.8%us, 1.3%sy, 0.0%ni, 0.0%id, 92.9%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu11 : 6.4%us, 1.4%sy, 0.0%ni, 92.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

If anyone has any suggestions on how to track this down I would appreciate it.

Nick D.

 

[thheo]

Leave "iostat 1" running for a while and see what device may cause high "tps" or just high transfer rate.
Or if you have some high network throughput with processing maybe you have network I/O.. you can also check /proc/interrupts
a few times in a row to see what device is triggered for interrupts..