Load spikes issue

I

itnocprimary

New Member
Dec 25, 2009
16
0
1
We had load spikes issue on our server having OS ubuntu 9.04 with kernel 2.6.24-7-pve and pve-manager 1.3-1. Load used to reache to 4.0-5.0 multiple times in a day, without any Virutal Container running on the server.

To resolve load spike issue, we have upraged the system to proxmox kernel 2.6.24-10-pve and pve-manager: 1.5-8, but still same issue persist, even though there was no Virtual Container running.

However, when we booted the system with the Ubuntu kernel 2.6.28-11-server the load average on the server is below 0.2.

Server is having 2 Intel Quad Core i7 Xeon processory (Hyperthreaded so the OS sees a total of 16 processors) and 72GB DDR3 Memory.

Please help me to figure out the issue and let me know if you need more information.
 
any backup tasks configured?
 
Please file top command output when load increased above to 2, without any Virutal Container running on the server.

top - 12:36:53 up 1 day, 8:00, 0 users, load average: 2.69, 0.92, 0.33
Tasks: 180 total, 4 running, 176 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 857568k used, 73364432k free, 167088k buffers
Swap: 5857272k total, 0k used, 5857272k free, 103468k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6388 root 20 0 93216 26m 2164 S 0 0.0 0:14.14 pvemirror
22326 root 20 0 88356 23m 2052 R 0 0.0 0:00.34 pvedaemon
1 root 20 0 5248 2036 632 S 0 0.0 0:02.18 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.04 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2

top - 12:37:54 up 1 day, 8:01, 0 users, load average: 2.25, 1.12, 0.43
Tasks: 192 total, 1 running, 191 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 863196k used, 73358804k free, 167104k buffers
Swap: 5857272k total, 0k used, 5857272k free, 103472k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6237 root 20 0 176m 27m 1652 S 2 0.0 0:32.93 console-kit-dae
1 root 20 0 5248 2036 632 S 0 0.0 0:02.18 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.04 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2

top - 18:40:00 up 1 day, 13:58, 0 users, load average: 2.13, 0.85, 0.31
Tasks: 189 total, 2 running, 187 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 868392k used, 73353608k free, 171240k buffers
Swap: 5857272k total, 0k used, 5857272k free, 105712k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6388 root 20 0 93216 26m 2164 S 0 0.0 0:16.88 pvemirror
31196 root 20 0 88356 23m 2052 S 0 0.0 0:00.88 pvedaemon
1 root 20 0 5248 2036 632 S 0 0.0 0:02.20 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2

top - 18:40:00 up 1 day, 13:58, 0 users, load average: 2.13, 0.85, 0.31
Tasks: 189 total, 2 running, 187 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 868392k used, 73353608k free, 171240k buffers
Swap: 5857272k total, 0k used, 5857272k free, 105712k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6388 root 20 0 93216 26m 2164 S 0 0.0 0:16.88 pvemirror
31196 root 20 0 88356 23m 2052 S 0 0.0 0:00.88 pvedaemon
1 root 20 0 5248 2036 632 S 0 0.0 0:02.20 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2

top - 18:41:35 up 1 day, 13:59, 0 users, load average: 2.39, 1.17, 0.47
Tasks: 197 total, 10 running, 186 sleeping, 0 stopped, 1 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 877208k used, 73344792k free, 171288k buffers
Swap: 5857272k total, 0k used, 5857272k free, 105780k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
59 root 15 -5 0 0 0 S 0 0.0 0:13.00 events/8
22326 root 20 0 88356 23m 2060 S 0 0.0 0:01.92 pvedaemon
6237 root 20 0 177m 30m 1652 R 0 0.0 0:38.19 console-kit-dae
1 root 20 0 5248 2036 632 S 0 0.0 0:02.20 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2

top - 18:43:01 up 1 day, 14:01, 0 users, load average: 2.09, 1.37, 0.60
Tasks: 200 total, 10 running, 190 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 879252k used, 73342748k free, 171296k buffers
Swap: 5857272k total, 0k used, 5857272k free, 105788k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6388 root 20 0 93216 26m 2164 S 0 0.0 0:16.92 pvemirror
5546 syslog 20 0 12380 752 564 S 0 0.0 0:00.90 syslogd
22326 root 20 0 88356 23m 2060 S 0 0.0 0:01.94 pvedaemon
1 root 20 0 5248 2036 632 S 0 0.0 0:02.20 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
 
More output is given below

top - 18:43:02 up 1 day, 14:01, 0 users, load average: 2.09, 1.37, 0.60
Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 870752k used, 73351248k free, 171316k buffers
Swap: 5857272k total, 0k used, 5857272k free, 105768k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6237 root 20 0 177m 30m 1652 S 2 0.0 0:38.24 console-kit-dae
1 root 20 0 5248 2036 632 S 0 0.0 0:02.20 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2
12 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/3

top - 18:43:02 up 1 day, 14:01, 0 users, load average: 2.09, 1.37, 0.60
Tasks: 202 total, 1 running, 201 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 871560k used, 73350440k free, 171316k buffers
Swap: 5857272k total, 0k used, 5857272k free, 105768k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6237 root 20 0 177m 30m 1652 S 2 0.0 0:38.24 console-kit-dae
1 root 20 0 5248 2036 632 S 0 0.0 0:02.20 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.06 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2
12 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/3

top - 08:31:01 up 2 days, 3:49, 0 users, load average: 2.53, 3.30, 1.59
Tasks: 181 total, 1 running, 180 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 918760k used, 73303240k free, 177448k buffers
Swap: 5857272k total, 0k used, 5857272k free, 145092k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6388 root 20 0 93216 26m 2164 S 4 0.0 0:23.34 pvemirror
31196 root 20 0 88356 23m 2060 S 4 0.0 0:05.42 pvedaemon
1 root 20 0 5248 2036 632 S 0 0.0 0:02.24 init
2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT -5 0 0 0 S 0 0.0 0:00.08 migration/0
4 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT -5 0 0 0 S 0 0.0 0:00.08 migration/1
7 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
10 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/2
11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2
 
itnocprimary said:
...
top - 12:36:53 up 1 day, 8:00, 0 users, load average: 2.69, 0.92, 0.33
Tasks: 180 total, 4 running, 176 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 74222000k total, 857568k used, 73364432k free, 167088k buffers
Swap: 5857272k total, 0k used, 5857272k free, 103468k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6388 root 20 0 93216 26m 2164 S 0 0.0 0:14.14 pvemirror
22326 root 20 0 88356 23m 2052 R 0 0.0 0:00.34 pvedaemon
...
Click to expand...
Hi,
i see no problem - the load value says how many processes are ready for using the cpu.
High loads can be occur if the disk to slow and many processes must wait for io.
In your case pvemirror and another task are doing something, but you have 16 cores. So only if your load higher than 16 a process must waiting for the cpu.

Load is one value, but you must look also for idle, system and wait. I think this values are pretty good on your hardware. You can try to stress your disks (e.g. with bonnie++) and look then to the values - also with running VMs.

Udo
 
Same time we have also generated SAR and MPSTAT report. We have found no iowait, idle or system. But still load increases.

[SAR]~~~~
Linux v02 2.6.24-10-pve #1 SMP PREEMPT Tue Dec 29 10:16:15 CET 2009 x86_64 06/02/2010
12:36:53 cpu %usr %nice %sys %irq %softirq %wait %idle _cpu_
12:37:53 all 0 0 0 0 0 0 100
0 0 0 0 0 0 0 100
1 0 0 0 0 0 0 100
2 0 0 0 0 0 0 100
3 0 0 0 0 0 0 100
4 0 0 0 0 0 0 100
5 0 0 0 0 0 0 100
6 0 0 0 0 0 0 100
7 0 0 0 0 0 0 100
8 0 0 0 0 0 0 100
9 0 0 0 0 0 0 100
10 0 0 0 0 0 0 100
11 0 0 0 0 0 0 100
12 0 0 0 0 0 0 100
13 0 0 0 0 0 0 100
14 0 0 0 0 0 0 100
15 0 0 0 0 0 0 100


~~[MPSTAT]~~~~
Linux 2.6.24-10-pve (v02) 06/02/10 _x86_64_ (16 CPU)
12:37:55 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
12:37:55 all 0.01 0.00 0.12 0.00 0.00 0.00 0.00 0.00 99.87
12:37:55 0 0.02 0.00 0.30 0.04 0.01 0.00 0.00 0.00 99.63
12:37:55 1 0.01 0.00 0.29 0.00 0.00 0.00 0.00 0.00 99.70
12:37:55 2 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.00 99.97
12:37:55 3 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.97
12:37:55 4 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98
12:37:55 5 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.99
12:37:55 6 0.01 0.00 0.26 0.00 0.00 0.00 0.00 0.00 99.73
12:37:55 7 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.99
12:37:55 8 0.01 0.01 0.28 0.00 0.00 0.00 0.00 0.00 99.70
12:37:55 9 0.00 0.00 0.27 0.00 0.00 0.00 0.00 0.00 99.72
12:37:55 10 0.03 0.00 0.05 0.00 0.00 0.00 0.00 0.00 99.92
12:37:55 11 0.01 0.00 0.04 0.00 0.00 0.00 0.00 0.00 99.95
12:37:55 12 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.00 99.97



[SAR]~~~~
Linux v02 2.6.24-10-pve #1 SMP PREEMPT Tue Dec 29 10:16:15 CET 2009 x86_64 06/02/2010
18:40:00 cpu %usr %nice %sys %irq %softirq %wait %idle _cpu_
18:40:01 all 1 0 5 0 0 0 94
0 9 0 9 0 0 0 82
1 0 0 0 0 0 0 100
2 0 0 18 0 0 0 82
3 0 0 0 0 0 0 100
4 0 0 9 0 0 0 91
5 0 0 0 0 0 0 100
6 0 0 0 0 0 0 100
7 0 0 0 0 0 0 100
8 9 0 18 0 0 0 73
9 0 0 9 0 0 0 91
10 0 0 9 0 0 0 91
11 0 0 0 0 0 0 100
12 0 0 9 0 0 0 91
13 0 0 0 0 0 0 100
14 0 0 0 0 0 0 100
15 0 0 0 0 0 0 100


~~[MPSTAT]~~~~

Linux 2.6.24-10-pve (v02) 06/02/10 _x86_64_ (16 CPU)
18:40:01 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
18:40:01 all 0.01 0.00 0.10 0.00 0.00 0.00 0.00 0.00 99.88
18:40:01 0 0.02 0.00 0.26 0.04 0.01 0.00 0.00 0.00 99.67
18:40:01 1 0.01 0.00 0.25 0.00 0.00 0.00 0.00 0.00 99.74
18:40:01 2 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.00 99.97
18:40:01 3 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98
18:40:01 4 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98
18:40:01 5 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.99
18:40:01 6 0.01 0.00 0.22 0.00 0.00 0.00 0.00 0.00 99.77
18:40:01 7 0.00 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.99
18:40:01 8 0.01 0.01 0.24 0.00 0.00 0.00 0.00 0.00 99.74
18:40:01 9 0.00 0.00 0.23 0.00 0.00 0.00 0.00 0.00 99.76
18:40:01 10 0.03 0.00 0.05 0.00 0.00 0.00 0.00 0.00 99.92
18:40:01 11 0.01 0.00 0.04 0.00 0.00 0.00 0.00 0.00 99.95
18:40:01 12 0.01 0.00 0.02 0.00 0.00 0.00 0.00 0.00 99.97
18:40:01 13 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.00 99.98
18:40:01 14 0.01 0.00 0.23 0.00 0.00 0.00 0.00 0.00 99.76
18:40:01 15 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.00 99.98
 
You must log in or register to reply here.
Top Bottom