Strange performance problem

gmckeown

Renowned Member
Nov 13, 2010
8
0
66
I have a server installed with the latest prox. It's not top of the line box.

About every 4-5 min. The I/O delay goes up to 50%+ for about 10-20 seconds, then back down again. The server has 16*300 GB sas drives in a raid 10, 32 GB Ram and 16 cores. We tried swapping out the raid card - no change. If I install Debian 7.8, or Ubuntu, or Cent, and pound the array with I/O, there is no delays at all.

Tried a bare-metal install and that did not solve the issue. The only VM running is a mysql slave that is replicating for HA.

CPU usage is 1-4%. Ram is 8GB out of 32.

pveperf shows:
CPU BOGOMIPS: 81065.52
REGEX/SECOND: 1045465
HD SIZE: 54.88 GB (/dev/disk/by-uuid/505ed783-87e8-435e-b0b8-775244613544)
BUFFERED READS: 354.39 MB/sec
AVERAGE SEEK TIME: 4.79 ms
FSYNCS/SECOND: 3479.59

I'm stumped. Does anybody have any ideas what we can try to isolate this?
 
I have a server installed with the latest prox. It's not top of the line box.

About every 4-5 min. The I/O delay goes up to 50%+ for about 10-20 seconds, then back down again. The server has 16*300 GB sas drives in a raid 10, 32 GB Ram and 16 cores. We tried swapping out the raid card - no change. If I install Debian 7.8, or Ubuntu, or Cent, and pound the array with I/O, there is no delays at all.

Tried a bare-metal install and that did not solve the issue. The only VM running is a mysql slave that is replicating for HA.

CPU usage is 1-4%. Ram is 8GB out of 32.

pveperf shows:
CPU BOGOMIPS: 81065.52
REGEX/SECOND: 1045465
HD SIZE: 54.88 GB (/dev/disk/by-uuid/505ed783-87e8-435e-b0b8-775244613544)
BUFFERED READS: 354.39 MB/sec
AVERAGE SEEK TIME: 4.79 ms
FSYNCS/SECOND: 3479.59

I'm stumped. Does anybody have any ideas what we can try to isolate this?
Hi,
you can try to use atop (apt-get install atop) to see, which process do the IO and perhaps see bottlenecks.

Udo