Out of memory!!

Amori · Aug 20, 2014

Hello,
I have 2 servers with same configurations. I got a lot of "out of out of memory" errors on my second server. Therefore the server force turn off VM's. please check my Syslog here
specially this line "Aug 20 00:24:29 sd-40376 kernel: Out of memory: OOM killed process 4850 (kvm) score 0 vm:4604284kB, rss:4111012kB, swap:0kB"

PHP:

Aug 20 00:24:29 sd-40376 kernel: ksm_stable_node      : size   59420672 objsize         48
Aug 20 00:24:29 sd-40376 kernel: radix_tree_node      : size    6463488 objsize        560
Aug 20 00:24:29 sd-40376 kernel: size-192             : size    5083136 objsize        192
Aug 20 00:24:29 sd-40376 kernel: dentry               : size    7036928 objsize        224
Aug 20 00:24:29 sd-40376 kernel: kvm_mmu_page_header  : size    5128192 objsize        176
Aug 20 00:24:29 sd-40376 kernel: ksm_rmap_item        : size  952262656 objsize         64
Aug 20 00:24:29 sd-40376 kernel: ext4_inode_cache     : size   16891904 objsize       1128
Aug 20 00:24:29 sd-40376 kernel: size-128             : size    6668288 objsize        128
Aug 20 00:24:29 sd-40376 kernel: nf_conntrack_ffffffff81b1d000: size   10063872 objsize        312
Aug 20 00:24:29 sd-40376 kernel: size-16384           : size   17319744 objsize      16384
Aug 20 00:24:29 sd-40376 kernel: OOM selected worst BC 0 (overdraft 1806644):
Aug 20 00:24:29 sd-40376 kernel: RAM: 7823617 / inf [0] SWAP: 154155 / inf [0] KMEM: 176345088 / inf [0] DCSZ: 26546153 / inf [0] OOMG: 7930292 / inf [26] Dirty 0 Wback 195 Dche 20080 Prnd 0
Aug 20 00:24:29 sd-40376 kernel: Out of memory: OOM killed process 4850 (kvm) score 0 vm:4604284kB, rss:4111012kB, swap:0kB
Aug 20 00:24:29 sd-40376 kernel: oom-killer in ub -1 generation 26 ends: task died
Aug 20 00:24:29 sd-40376 kernel: Mem-Info:
Aug 20 00:24:29 sd-40376 kernel: Node 0 DMA per-cpu:
Aug 20 00:24:29 sd-40376 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    2: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    3: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    4: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    5: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    6: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    7: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    8: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    9: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU   10: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU   11: hi:    0, btch:   1 usd:   0
Aug 20 00:24:29 sd-40376 kernel: Node 0 DMA32 per-cpu:
Aug 20 00:24:29 sd-40376 kernel: CPU    0: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    1: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    2: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    3: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    4: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    5: hi:  186, btch:  31 usd: 185
Aug 20 00:24:29 sd-40376 kernel: CPU    6: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    7: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    8: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU    9: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU   10: hi:  186, btch:  31 usd:   0
Aug 20 00:24:29 sd-40376 kernel: CPU   11: hi:  186, btch:  31 usd:   1
Aug 20 00:24:29 sd-40376 kernel: Node 0 Normal per-cpu:
Aug 20 00:24:29 sd-40376 kernel: CPU    0: hi:  186, btch:  31 usd:  34
Aug 20 00:24:29 sd-40376 kernel: CPU    1: hi:  186, btch:  31 usd:  16
Aug 20 00:24:29 sd-40376 kernel: CPU    2: hi:  186, btch:  31 usd:  22
Aug 20 00:24:29 sd-40376 kernel: CPU    3: hi:  186, btch:  31 usd:  28
Aug 20 00:24:29 sd-40376 kernel: CPU    4: hi:  186, btch:  31 usd:  33
Aug 20 00:24:29 sd-40376 kernel: CPU    5: hi:  186, btch:  31 usd: 164
Aug 20 00:24:29 sd-40376 kernel: CPU    6: hi:  186, btch:  31 usd:  38
Aug 20 00:24:29 sd-40376 kernel: CPU    7: hi:  186, btch:  31 usd:  41
Aug 20 00:24:29 sd-40376 kernel: CPU    8: hi:  186, btch:  31 usd:  21
Aug 20 00:24:29 sd-40376 kernel: CPU    9: hi:  186, btch:  31 usd:  36
Aug 20 00:24:29 sd-40376 kernel: CPU   10: hi:  186, btch:  31 usd:   9
Aug 20 00:24:29 sd-40376 kernel: CPU   11: hi:  186, btch:  31 usd:  24

PHP:

proxmox-ve-2.6.32: 3.2-129 (running kernel: 2.6.32-30-pve)pve-manager: 3.2-4 (running version: 3.2-4/e24a91c1)pve-kernel-2.6.32-30-pve: 2.6.32-129lvm2: 2.02.98-pve4clvm: 2.02.98-pve4corosync-pve: 1.4.5-1openais-pve: 1.1.4-3libqb0: 0.11.1-2redhat-cluster-pve: 3.2.0-2resource-agents-pve: 3.9.2-4fence-agents-pve: 4.0.5-1pve-cluster: 3.0-12qemu-server: 3.1-16pve-firmware: 1.1-3libpve-common-perl: 3.0-18libpve-access-control: 3.0-11libpve-storage-perl: 3.0-19pve-libspice-server1: 0.12.4-3vncterm: 1.1-6vzctl: 4.0-1pve5vzprocps: 2.0.11-2vzquota: 3.1-2pve-qemu-kvm: 1.7-8ksm-control-daemon: 1.1-1glusterfs-client: 3.4.2-1

Any solution on this please?

udo · Aug 20, 2014

Amori said:
Hello,
I have 2 servers with same configurations. I got a lot of "out of out of memory" errors on my second server. Therefore the server force turn off VM's.
...
Any solution on this please?

Hi,
look for used memmory - perhaps you have an process with an memory leak (can be an VM).

Use "top" and press "m".

Look also the output of "free".

Udo

Amori · Aug 20, 2014

udo said:
Hi,
look for used memmory - perhaps you have an process with an memory leak (can be an VM).

Use "top" and press "m".

Look also the output of "free".

Udo

Hello,

Thanks for replay, I did not understand use "top" and press "m". Could you please explain more..

Here is the outpot of free

free
total used free shared buffers cached
Mem: 32900712 30006080 2894632 0 61132 4280784
-/+ buffers/cache: 25664164 7236548
Swap: 999992 953708 46284

udo · Aug 20, 2014

Amori said:
Hello,

Thanks for replay, I did not understand use "top" and press "m". Could you please explain more..

Hi,
see "man top"

Code:

man top
...
 Startup Defaults
       The  following startup defaults assume no configuration file, thus no user customizations.  Even so, items shown with an asterisk (´*´) could be overrid‐
       den through the command-line.  All are explained in detail in the sections that follow.

           Global-defaults
              'A' - Alt display      Off (full-screen)
            * 'd' - Delay time       3.0 seconds
            * 'H' - Threads mode     Off (summarize as tasks)
              'I' - Irix mode        On  (no, 'solaris' smp)
            * 'p' - PID monitoring   Off (show all processes)
            * 's' - Secure mode      Off (unsecured)
              'B' - Bold enable      On  (yes, bold globally)
           Summary-Area-defaults
              'l' - Load Avg/Uptime  On  (thus program name)
              't' - Task/Cpu states  On  (1+1 lines, see '1')
             [B] 'm' - Mem/Swap usage   On  (2 lines worth)[/B]
              '1' - Single Cpu       On  (thus 1 line if smp)

inside top "m" sort for memory usage!

Udo

Amori · Aug 20, 2014

Got it, here is the outpot

top - 13:20:40 up 60 days, 14:44, 3 users, load average: 2,31, 2,68, 2,83
Tasks: 326 total, 1 running, 325 sleeping, 0 stopped, 0 zombie
%Cpu(s): 41,8 us, 8,7 sy, 0,0 ni, 48,6 id, 0,0 wa, 0,0 hi, 0,9 si, 0,0 st
KiB Mem: 32900712 total, 29644040 used, 3256672 free, 66932 buffers
KiB Swap: 999992 total, 957312 used, 42680 free, 4032928 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1018788 root 20 0 3466m 3,0g 1444 S 211 9,6 3346:44 kvm
612505 root 20 0 2510m 2,0g 1612 S 106 6,4 9572:39 kvm
797383 root 20 0 3471m 3,0g 1528 S 93 9,6 3117:41 kvm
495255 root 20 0 3520m 3,0g 1632 S 43 9,6 61624:05 kvm
502913 root 20 0 3522m 2,9g 1536 S 32 9,2 22091:34 kvm
142 root 25 5 0 0 0 S 30 0,0 12947:17 ksmd
5531 root 20 0 4560m 3,9g 1548 S 21 12,5 314:19.51 kvm
23960 root 20 0 4559m 3,9g 1564 S 19 12,5 189:46.59 kvm
794999 root 20 0 2442m 2,0g 1512 S 15 6,4 923:39.80 kvm
825954 root 20 0 3462m 2,8g 1536 S 14 9,1 6759:00 kvm
981188 root 20 0 3520m 2,8g 1596 S 10 8,9 2872:57 kvm
4884 root 20 0 4496m 3,9g 1548 S 5 12,5 68:15.77 kvm
46092 root 20 0 4496m 3,9g 1596 S 5 12,5 34:57.05 kvm
46136 root 20 0 4496m 3,9g 1640 S 5 12,5 40:34.59 kvm
921168 root 20 0 2496m 1,9g 1520 S 5 6,0 8953:39 kvm
5346 root 20 0 4484m 3,9g 1460 S 4 12,5 63:44.11 kvm
46933 root 20 0 4494m 3,9g 1788 S 4 12,5 33:15.99 kvm
890883 root 20 0 3396m 2,9g 1444 S 4 9,2 197:42.81 kvm
84435 root 20 0 23396 1948 1176 R 2 0,0 0:00.12 top
84313 www-data 20 0 280m 21m 3536 S 1 0,1 0:00.18 pveproxy worker
1 root 20 0 10604 640 604 S 0 0,0 0:29.83 init
2 root 20 0 0 0 0 S 0 0,0 0:00.20 kthreadd
3 root rt 0 0 0 0 S 0 0,0 0:43.18 migration/0
4 root 20 0 0 0 0 S 0 0,0 10:57.62 ksoftirqd/0
5 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/0
6 root rt 0 0 0 0 S 0 0,0 0:03.94 watchdog/0
7 root rt 0 0 0 0 S 0 0,0 0:34.15 migration/1
8 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0 0,0 2:37.69 ksoftirqd/1
10 root rt 0 0 0 0 S 0 0,0 0:02.38 watchdog/1
11 root rt 0 0 0 0 S 0 0,0 0:31.34 migration/2
12 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/2
13 root 20 0 0 0 0 S 0 0,0 1:55.74 ksoftirqd/2
14 root rt 0 0 0 0 S 0 0,0 0:02.34 watchdog/2
15 root rt 0 0 0 0 S 0 0,0 0:30.17 migration/3
16 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/3
17 root 20 0 0 0 0 S 0 0,0 1:33.67 ksoftirqd/3
18 root rt 0 0 0 0 S 0 0,0 0:02.29 watchdog/3
19 root rt 0 0 0 0 S 0 0,0 0:21.87 migration/4
20 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/4
21 root 20 0 0 0 0 S 0 0,0 1:22.10 ksoftirqd/4
22 root rt 0 0 0 0 S 0 0,0 0:02.29 watchdog/4
23 root rt 0 0 0 0 S 0 0,0 0:19.05 migration/5
24 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/5
25 root 20 0 0 0 0 S 0 0,0 1:11.54 ksoftirqd/5
26 root rt 0 0 0 0 S 0 0,0 0:02.35 watchdog/5
27 root rt 0 0 0 0 S 0 0,0 0:35.56 migration/6
28 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/6
29 root 20 0 0 0 0 S 0 0,0 3:59.17 ksoftirqd/6
30 root rt 0 0 0 0 S 0 0,0 0:02.48 watchdog/6
31 root rt 0 0 0 0 S 0 0,0 0:40.74 migration/7
32 root rt 0 0 0 0 S 0 0,0 0:00.00 migration/7
33 root 20 0 0 0 0 S 0 0,0 1:32.52 ksoftirqd/7
34 root rt 0 0 0 0 S 0 0,0 0:02.34 watchdog/7
35 root rt 0 0 0 0 S 0 0,0 0:30.03 migration/8

udo said:

Hi,
see "man top"

Code:

man top
...
 Startup Defaults
       The  following startup defaults assume no configuration file, thus no user customizations.  Even so, items shown with an asterisk (´*´) could be overrid‐
       den through the command-line.  All are explained in detail in the sections that follow.

           Global-defaults
              'A' - Alt display      Off (full-screen)
            * 'd' - Delay time       3.0 seconds
            * 'H' - Threads mode     Off (summarize as tasks)
              'I' - Irix mode        On  (no, 'solaris' smp)
            * 'p' - PID monitoring   Off (show all processes)
            * 's' - Secure mode      Off (unsecured)
              'B' - Bold enable      On  (yes, bold globally)
           Summary-Area-defaults
              'l' - Load Avg/Uptime  On  (thus program name)
              't' - Task/Cpu states  On  (1+1 lines, see '1')
             [B] 'm' - Mem/Swap usage   On  (2 lines worth)[/B]
              '1' - Single Cpu       On  (thus 1 line if smp)

inside top "m" sort for memory usage!

Udo

udo · Aug 20, 2014

Amori said:
Got it, here is the outpot

top - 13:20:40 up 60 days, 14:44, 3 users, load average: 2,31, 2,68, 2,83
Tasks: 326 total, 1 running, 325 sleeping, 0 stopped, 0 zombie
%Cpu(s): 41,8 us, 8,7 sy, 0,0 ni, 48,6 id, 0,0 wa, 0,0 hi, 0,9 si, 0,0 st
KiB Mem: 32900712 total, 29644040 used, 3256672 free, 66932 buffers
KiB Swap: 999992 total, 957312 used, 42680 free, 4032928 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1018788 root 20 0 3466m 3,0g 1444 S 211 9,6 3346:44 kvm
612505 root 20 0 2510m 2,0g 1612 S 106 6,4 9572:39 kvm
797383 root 20 0 3471m 3,0g 1528 S 93 9,6 3117:41 kvm
495255 root 20 0 3520m 3,0g 1632 S 43 9,6 61624:05 kvm
502913 root 20 0 3522m 2,9g 1536 S 32 9,2 22091:34 kvm
142 root 25 5 0 0 0 S 30 0,0 12947:17 ksmd
5531 root 20 0 4560m 3,9g 1548 S 21 12,5 314:19.51 kvm
23960 root 20 0 4559m 3,9g 1564 S 19 12,5 189:46.59 kvm
794999 root 20 0 2442m 2,0g 1512 S 15 6,4 923:39.80 kvm
825954 root 20 0 3462m 2,8g 1536 S 14 9,1 6759:00 kvm
981188 root 20 0 3520m 2,8g 1596 S 10 8,9 2872:57 kvm
4884 root 20 0 4496m 3,9g 1548 S 5 12,5 68:15.77 kvm
46092 root 20 0 4496m 3,9g 1596 S 5 12,5 34:57.05 kvm
46136 root 20 0 4496m 3,9g 1640 S 5 12,5 40:34.59 kvm
921168 root 20 0 2496m 1,9g 1520 S 5 6,0 8953:39 kvm
5346 root 20 0 4484m 3,9g 1460 S 4 12,5 63:44.11 kvm
46933 root 20 0 4494m 3,9g 1788 S 4 12,5 33:15.99 kvm
890883 root 20 0 3396m 2,9g 1444 S 4 9,2 197:42.81 kvm
84435 root 20 0 23396 1948 1176 R 2 0,0 0:00.12 top

Hi,
you have "only" 32GB RAM but 7 VMs with app. 3GB + 7 VMs with app. 4GB + 2 VMs with app. 2GB which is round 52GB RAM.

And then you are wounder about memory problems??

Udo

Amori · Aug 20, 2014

udo said:
Hi,
you have "only" 32GB RAM but 7 VMs with app. 3GB + 7 VMs with app. 4GB + 2 VMs with app. 2GB which is round 52GB RAM.

And then you are wounder about memory problems??

Udo

It was not an issue on my first server. I run like 23 vms with 2GB+ RAM per vm.

e100 · Aug 20, 2014

Amori said:
It was not an issue on my first server. I run like 23 vms with 2GB+ RAM per vm.

If those 23 VM don't actually use the RAM KSM will merge pages since they are all zeros.
If those 23 VM are running the same OS and kernel, KSM helps again.

Start using lots of RAM in the VMs that results in pages that KSM cannot merge, you run out of RAM.

Your other server has the potential to have the same problem:
23 * 2GB = 46GB > 32GB
Just because it has not had an issue does not mean it never will.

adamb · Aug 20, 2014

e100 said:
If those 23 VM don't actually use the RAM KSM will merge pages since they are all zeros.
If those 23 VM are running the same OS and kernel, KSM helps again.

Start using lots of RAM in the VMs that results in pages that KSM cannot merge, you run out of RAM.

Your other server has the potential to have the same problem:
23 * 2GB = 46GB > 32GB
Just because it has not had an issue does not mean it never will.

Well said, KSM can only do so much. Your pushing the limits of that hardware for sure.

Amori · Aug 26, 2014

I count the RAM my second Server use for all my VPS, about 72 GB RAM. Yes the VPS are all same type, Windows VPS. The second server never turn off the VPS by it self when its out of memory. Is there any way to disable that option?

@udo @e100 @adamb

Amori · Aug 26, 2014

I count the RAM my second Server use for all my VPS, about 72 GB RAM. Yes the VPS are all same type, Windows VPS. The second server never turn off the VPS by it self when its out of memory. Is there any way to disable that option?

udo said:
Udo

e100 said:
Udo

adamb said:
Udo

e100 · Aug 28, 2014

You could try searching google for "disable OOM killer"

Disabling the OOM killer will not solve your problem only cause other issues, like the OS in your VM crashing because it wants to use RAM that does not exist or you being unable to log in because there is no free ram.
If you disable it and run out of RAM expect a worse situation than a few killed VMs.

The OOM killer is your friend, it will kill off enough VMs to keep THE REST of them running stable.
If the OOM killer is killing VPS's because you are out of memory then you need to run fewer VPS or get more RAM.

Its like trying to fit 72lbs of potatoes into a sack designed for 32lbs. Sure the sack might stretch a little but eventually it will burst apart at the seams and your potatoes will fall out.

RRJ · Aug 28, 2014

Come on, do not believe in miracles and just remember one simple rule: NEVER EVER over-commit your physical RAM and you'll be happy. By the way, second rule could be: NEVER EVER disable OOM killer.

and just once more again:

Just because it has not had an issue does not mean it never will.

(c) e100.

He's absolutely right.

You've been just so lucky to get 72 GBs of your 32.

p.s
RAMs are so cheap.. add some

p.p.s
I would agree, that killing few VMs is much more acceptable solution than died root's bash session ie.

Search

Search

Out of memory!!

Amori

Active Member

udo

Distinguished Member

Amori

Active Member

udo

Distinguished Member

Amori

Active Member

udo

Distinguished Member

Amori

Active Member

e100

Renowned Member

adamb

Famous Member

Amori

Active Member

Amori

Active Member

e100

Renowned Member

RRJ

Member