1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Out of Memory & Very very high load generated by CT

Discussion in 'Proxmox VE: Installation and configuration' started by marcinc81, Dec 7, 2017.

Tags:
  1. marcinc81

    marcinc81 New Member

    Joined:
    Dec 7, 2017
    Messages:
    2
    Likes Received:
    0
    Hi there,

    I'm using Proxmox 4.4 as it's the best virtual environment for my needs. Recently I discovered a reason of very high load on host: out of memory generated by CT. Basically some java process running in CT eats lot of memory and entire system hangs for a couple minutes, then back to normal. Load is more than 600-700, the system is not responding then.

    Here's /var/log/messages captured during (rather slightly before) high load:

    Code:
    Dec  7 11:37:47 vs3 kernel: [312709.328876] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
    Dec  7 11:37:47 vs3 kernel: [312709.328877]  0000000000000286 4d1810e2eb6729d4 ffff8814509ffc90 ffffffff813fcc63
    Dec  7 11:37:47 vs3 kernel: [312709.328878]  ffff8814509ffd68 ffff881c1766dc00 ffff8814509ffcf8 ffffffff8120c3cb
    Dec  7 11:37:47 vs3 kernel: [312709.328880]  0000000000000000 ffff881df487db40 ffff881e55741c00 ffff8814509ffce8
    Dec  7 11:37:47 vs3 kernel: [312709.328881] Call Trace:
    Dec  7 11:37:47 vs3 kernel: [312709.328885]  [<ffffffff813fcc63>] dump_stack+0x63/0x90
    Dec  7 11:37:47 vs3 kernel: [312709.328888]  [<ffffffff8120c3cb>] dump_header+0x67/0x1d5
    Dec  7 11:37:47 vs3 kernel: [312709.328891]  [<ffffffff81394f0a>] ? apparmor_capable+0x1aa/0x1b0
    Dec  7 11:37:47 vs3 kernel: [312709.328893]  [<ffffffff81192ba5>] oom_kill_process+0x205/0x3c0
    Dec  7 11:37:47 vs3 kernel: [312709.328896]  [<ffffffff811fff2f>] ? mem_cgroup_iter+0x1cf/0x380
    Dec  7 11:37:47 vs3 kernel: [312709.328898]  [<ffffffff81201ef8>] mem_cgroup_out_of_memory+0x2a8/0x2f0
    Dec  7 11:37:47 vs3 kernel: [312709.328899]  [<ffffffff81202c97>] mem_cgroup_oom_synchronize+0x347/0x360
    Dec  7 11:37:47 vs3 kernel: [312709.328901]  [<ffffffff811fdcc0>] ? mem_cgroup_begin_page_stat+0x90/0x90
    Dec  7 11:37:47 vs3 kernel: [312709.328902]  [<ffffffff811932a4>] pagefault_out_of_memory+0x44/0xc0
    Dec  7 11:37:47 vs3 kernel: [312709.328905]  [<ffffffff8106af7f>] mm_fault_error+0x7f/0x160
    Dec  7 11:37:47 vs3 kernel: [312709.328906]  [<ffffffff8106b783>] __do_page_fault+0x3e3/0x410
    Dec  7 11:37:47 vs3 kernel: [312709.328909]  [<ffffffff81003885>] ? syscall_trace_enter_phase1+0xc5/0x140
    Dec  7 11:37:47 vs3 kernel: [312709.328910]  [<ffffffff8106b7d2>] do_page_fault+0x22/0x30
    Dec  7 11:37:47 vs3 kernel: [312709.328912]  [<ffffffff81869038>] page_fault+0x28/0x30
    Dec  7 11:37:47 vs3 kernel: [312709.328913] Task in /lxc/1001112/ns killed as a result of limit of /lxc/1001112
    Dec  7 11:37:47 vs3 kernel: [312709.328917] memory: usage 1048576kB, limit 1048576kB, failcnt 0
    Dec  7 11:37:47 vs3 kernel: [312709.328918] memory+swap: usage 1048576kB, limit 1048576kB, failcnt 59092724
    Dec  7 11:37:47 vs3 kernel: [312709.328918] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
    Dec  7 11:37:47 vs3 kernel: [312709.328919] Memory cgroup stats for /lxc/1001112: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
    Dec  7 11:37:47 vs3 kernel: [312709.328925] Memory cgroup stats for /lxc/1001112/ns: cache:18168KB rss:1030408KB rss_huge:0KB mapped_file:5052KB dirty:36KB writeback:44KB swap:0KB inactive_anon:524188KB active_anon:524008KB inactive_file:0KB active_file:0KB unevictable:0KB
    Dec  7 11:37:47 vs3 kernel: [312709.328932] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
    Dec  7 11:37:47 vs3 kernel: [312709.329931] [20556]     0 20556     9324      319      22       3        0             0 systemd
    Dec  7 11:37:47 vs3 kernel: [312709.329933] [20801]     0 20801    11906      121      27       3        0             0 rpcbind
    Dec  7 11:37:47 vs3 kernel: [312709.329935] [20807]   106 20807    45147      129      24       3        0             0 rsyslogd
    Dec  7 11:37:47 vs3 kernel: [312709.329937] [20809]     0 20809     6516       66      17       3        0             0 cron
    Dec  7 11:37:47 vs3 kernel: [312709.329938] [20851]     0 20851     3210       36      12       3        0             0 agetty
    Dec  7 11:37:47 vs3 kernel: [312709.329941] [20852]     0 20852     3210       35      12       3        0             0 agetty
    Dec  7 11:37:47 vs3 kernel: [312709.329951] [20854]     0 20854     3210       35      12       3        0             0 agetty
    Dec  7 11:37:47 vs3 kernel: [312709.329953] [20899]     0 20899    22428      235      41       3        0             0 saslauthd
    Dec  7 11:37:47 vs3 kernel: [312709.329954] [20900]     0 20900    22428      239      41       3        0             0 saslauthd
    Dec  7 11:37:47 vs3 kernel: [312709.329957] [24565]     0 24565      255       21       4       2        0             0 AccountingServe
    Dec  7 11:37:47 vs3 kernel: [312709.329958] [24995]     0 24995   396041     4390      60       4        0             0 ts3server_linux
    Dec  7 11:37:47 vs3 kernel: [312709.330018] [11283]     0 11283     6068      118      16       3        0             0 screen
    Dec  7 11:37:47 vs3 kernel: [312709.330020] [11284]     0 11284  1024376    69215     225       7        0             0 java
    Dec  7 11:37:47 vs3 kernel: [312709.330036] [24015]     0 24015     8818     1144      20       3        0             0 systemd-journal
    Dec  7 11:37:47 vs3 kernel: [312709.330055] [17811]     0 17811    17898      358      39       3        0             0 apache2
    Dec  7 11:37:47 vs3 kernel: [312709.330057] [18858]     0 18858    16380      181      35       3        0         -1000 sshd
    Dec  7 11:37:47 vs3 kernel: [312709.330058] [18991]    33 18991   139377      479      71       3        0             0 apache2
    Dec  7 11:37:47 vs3 kernel: [312709.330060] [19201]     0 19201     3764       48      13       3        0             0 xinetd
    Dec  7 11:37:47 vs3 kernel: [312709.330062] [19378]     0 19378    16352      118      22       3        0             0 master
    Dec  7 11:37:47 vs3 kernel: [312709.330063] [19380]   108 19380    16881      112      24       3        0             0 qmgr
    Dec  7 11:37:47 vs3 kernel: [312709.330065] [20512]     0 20512    25926      163      20       3        0             0 ntpd
    Dec  7 11:37:47 vs3 kernel: [312709.330066] [25135]  1001 25135     6069      119      16       3        0             0 screen
    Dec  7 11:37:47 vs3 kernel: [312709.330068] [25138]  1001 25138    14130      181      23       3        0             0 sinusbot
    Dec  7 11:37:47 vs3 kernel: [312709.330069] [25139]  1001 25139   338314     9646     112       6        0             0 sinusbot
    Dec  7 11:37:47 vs3 kernel: [312709.330074] [ 5082]  1001  5082   296249     2205     110       4        0             0 ts3client_linux
    Dec  7 11:37:47 vs3 kernel: [312709.330090] [32142]  1001 32142   287213     2344      99       4        0             0 ts3client_linux
    Dec  7 11:37:47 vs3 kernel: [312709.330146] [29070]   108 29070    16869      111      25       3        0             0 pickup
    Dec  7 11:37:47 vs3 kernel: [312709.330169] [  904]     0   904    23264      227      48       3        0             0 sshd
    Dec  7 11:37:47 vs3 kernel: [312709.330171] [ 2665]     0  2665     4566      128      14       3        0             0 bash
    Dec  7 11:37:47 vs3 kernel: [312709.330194] [26574]     0 26574     6069      118      16       3        0             0 screen
    Dec  7 11:37:47 vs3 kernel: [312709.330196] [26583]     0 26575  1171519   155512     406       8        0             0 java
    Dec  7 11:37:47 vs3 kernel: [312709.330206] [ 5490]     0  5490     6069       76      17       3        0             0 screen
    Dec  7 11:37:47 vs3 kernel: [312709.330208] [ 5491]     0  5491   775953     7849      78       6        0             0 java
    Dec  7 11:37:47 vs3 kernel: [312709.330209] [ 5541]     0  5541     6069       76      16       3        0             0 screen
    Dec  7 11:37:47 vs3 kernel: [312709.330211] [ 5542]     0  5542   762079     2420      68       6        0             0 java
    
    No users' processes in top, all cores are busy processing kernel tasks - haven't captured what exactly they do, because system was not responding.

    Does anyone encountered similar problem? Can't find any info regarding this issue. This problem maybe is not critical at the moment, but it happens from time to time - recently more often. I can't assign more RAM to the container. I would like to avoid hanging and just kill processes utilizing so much RAM (and it happen).

    Thanks in advance for any hints!
     
  2. czechsys

    czechsys Member

    Joined:
    Nov 18, 2015
    Messages:
    100
    Likes Received:
    2
    Tune Java. Limit memory for Java. Tune OOM killer.

    Problem solved.
     
    marcinc81 likes this.
  3. marcinc81

    marcinc81 New Member

    Joined:
    Dec 7, 2017
    Messages:
    2
    Likes Received:
    0
    Wow, didn't know it's that simple and well known problem! Can you advice what OOM 'switch' causes this behavior? Many thanks bro! :)
     
    #3 marcinc81, Dec 7, 2017
    Last edited: Dec 7, 2017

Share This Page