Performace issue

[jazzl0ver]

Hi,

I have 2 supermicro servers with latest Proxmox 3.0 installed on both: N1 - H8DMR-82 (nVidia chipset, AMD CPU) and N2 - X8SIE (Intel chipset, Xeon CPU).

While N2 works great, N1 has some performance issues: top shows ksoftirqd eats cpu. I booted N1 in a single user mode and simulated the load with dd:

# dd if=/dev/zero of=./testf bs=1M oflag=direct
^C1080+0 records in
1080+0 records out
1132462080 bytes (1.1 GB) copied, 7.8643 s, 144 MB/s

top - 15:13:12 up 44 min,  2 users,  load average: 0.08, 0.02, 0.01
Tasks: 136 total,   2 running, 134 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  3.2 sy,  0.0 ni, 75.1 id, 21.6 wa,  0.1 hi,  0.0 si,  0.0 st
KiB Mem:   8171412 total,   199068 used,  7972344 free,    11980 buffers
KiB Swap:        0 total,        0 used,        0 free,    97296 cached

    PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
     13 root      20   0     0    0    0 S  34.2  0.0   0:32.90 ksoftirqd/2
   5795 root      20   0 10852 1720  572 R  10.6  0.0   0:00.53 dd
    346 root      20   0     0    0    0 S   3.3  0.0   0:02.87 md10_raid10

Running the same command on N2 shows all's good (no VMs were stopped):

# dd if=/dev/zero of=./testf bs=1M oflag=direct
^C1924+0 records in
1924+0 records out
2017460224 bytes (2.0 GB) copied, 8.82385 s, 229 MB/s

top - 14:49:47 up 12 days,  3:48,  2 users,  load average: 0.08, 0.02, 0.01
Tasks: 250 total,   1 running, 249 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.4 us,  3.2 sy,  0.0 ni, 83.1 id, 11.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  32936236 total, 32739768 used,   196468 free,    30232 buffers
KiB Swap:        0 total,        0 used,        0 free, 25161156 cached

    PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
   4180 root      20   0 2381m 2.0g 1500 S  25.3  6.4   4435:55 kvm
 394205 root      20   0 3376m 3.0g 1528 S  12.6  9.4   1539:59 kvm
 398769 root      20   0 1351m 1.0g 1448 S   6.3  3.2 862:56.27 kvm
 541390 root      20   0 23260 1760 1180 R   6.3  0.0   0:00.28 top
 541416 root      20   0 10852 1732  588 D   6.3  0.0   0:00.49 dd

The only explanation I'm thinking of is the sata controller module: N1 uses sata_nv, N2 uses ahci.

Can somebody give me an advice how to eliminate the issue?

 

[hotwired007]

Do you have a different SATA controller you can use in N1?

 

[jazzl0ver]

Nope. Do you think it's the only possible way?

 

[hotwired007]

Nope. Do you think it's the only possible way?

it would either prove or disprove that its the SATA controller to blame - Proxmox tech support say to use server grade hardware - i doubt that the nvidia SATA controller qualifies

 

[jazzl0ver]

Good point. I will try to find something and get back with a result.

H8DMR-82 - is definitely a server grade hardware: http://www.supermicro.com/Aplus/motherboard/Opteron2000/MCP55/H8DMR-82.cfm

It might be just an old one.

 

[spirit]

Good point. I will try to find something and get back with a result.

H8DMR-82 - is definitely a server grade hardware: http://www.supermicro.com/Aplus/motherboard/Opteron2000/MCP55/H8DMR-82.cfm

It might be just an old one.

nVidia MCP55 Pro is not a server grade raid controller, it's just a fake soft raid controller

http://forum.proxmox.com/threads/8341-NVIDIA-MCP55-Pro-RAID

So don't expect too much performance from it.

 

[jazzl0ver]

I don't use it as a raid controller. It's just a SATA controller. Software raid rules
Anyway, trying to find another controller for tests.

 

[jazzl0ver]

Found two SATA controllers: Si3114 and MegaRAID SATA 150-4.
Si3114 performs even worse than nVidia's.
MegaRAID - performance is very bad as well:

# dd if=/dev/zero of=./testf bs=1M oflag=direct
^C433+0 records in
433+0 records out
454033408 bytes (454 MB) copied, 35.3743 s, 12.8 MB/

I think I need to find a modern controller..

 

[spirit]

Found two SATA controllers: Si3114 and MegaRAID SATA 150-4.
Si3114 performs even worse than nVidia's.
MegaRAID - performance is very bad as well:

# dd if=/dev/zero of=./testf bs=1M oflag=direct
^C433+0 records in
433+0 records out
454033408 bytes (454 MB) copied, 35.3743 s, 12.8 MB/

I think I need to find a modern controller..

note that if you use dd oflag=direct, you bypass the cache of the controller (if you have a controller with cache of course) , so writes are slower.

(it's like you force cache=directsync in your vms disk)

But indeed, that seem very slow for a 1M block size ...even for an old lsi megaraid controller.

 

[jazzl0ver]

Some news.

I've tried to run the test under another OS - CentOS 6. And there was no ksoftirqd eating CPU!! Although, the i/o wait is still around 20%, the absence of ksoftirqd in top says there's probably something in the Proxmox kernel. Any ideas what might it be?

 

[spirit]

maybe can you try last pve-kernel in pvetest repository, it's based on rhel 6.4 instead 6.3. (kernel should be the same than centos)

 

[sumsum]

Have you tried to set the I/O scheduler of the KVM guest to NOOP ? Just in case you run a kvm.

 

[jazzl0ver]

spirit, thank you very much!

Kernel from pvtest repo has fixed the issue:

top - 13:01:40 up 4 min,  1 user,  load average: 0.26, 0.16, 0.07
Tasks: 135 total,   1 running, 134 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  1.5 sy,  0.0 ni, 75.1 id, 23.3 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   8171168 total,   157612 used,  8013556 free,     3744 buffers
KiB Swap:        0 total,        0 used,        0 free,    81876 cached

    PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
   3792 root      20   0 10852 1716  572 D   6.0  0.0   0:01.08 dd if=/dev/zero of=./testf bs=1M oflag=direct
      1 root      20   0 10604  868  716 S   0.0  0.0   0:01.22 init [S]

 

[jazzl0ver]

sumsum, changing scheduler to cfq did not affect ksoftirqd issue. Anyway, thanks for your help!