Hi there,
I'm using Proxmox 4.4 as it's the best virtual environment for my needs. Recently I discovered a reason of very high load on host: out of memory generated by CT. Basically some java process running in CT eats lot of memory and entire system hangs for a couple minutes, then back to normal. Load is more than 600-700, the system is not responding then.
Here's /var/log/messages captured during (rather slightly before) high load:
No users' processes in top, all cores are busy processing kernel tasks - haven't captured what exactly they do, because system was not responding.
Does anyone encountered similar problem? Can't find any info regarding this issue. This problem maybe is not critical at the moment, but it happens from time to time - recently more often. I can't assign more RAM to the container. I would like to avoid hanging and just kill processes utilizing so much RAM (and it happen).
Thanks in advance for any hints!
I'm using Proxmox 4.4 as it's the best virtual environment for my needs. Recently I discovered a reason of very high load on host: out of memory generated by CT. Basically some java process running in CT eats lot of memory and entire system hangs for a couple minutes, then back to normal. Load is more than 600-700, the system is not responding then.
Here's /var/log/messages captured during (rather slightly before) high load:
Code:
Dec 7 11:37:47 vs3 kernel: [312709.328876] Hardware name: Supermicro Super Server/X10SRi-F, BIOS 2.0 12/17/2015
Dec 7 11:37:47 vs3 kernel: [312709.328877] 0000000000000286 4d1810e2eb6729d4 ffff8814509ffc90 ffffffff813fcc63
Dec 7 11:37:47 vs3 kernel: [312709.328878] ffff8814509ffd68 ffff881c1766dc00 ffff8814509ffcf8 ffffffff8120c3cb
Dec 7 11:37:47 vs3 kernel: [312709.328880] 0000000000000000 ffff881df487db40 ffff881e55741c00 ffff8814509ffce8
Dec 7 11:37:47 vs3 kernel: [312709.328881] Call Trace:
Dec 7 11:37:47 vs3 kernel: [312709.328885] [<ffffffff813fcc63>] dump_stack+0x63/0x90
Dec 7 11:37:47 vs3 kernel: [312709.328888] [<ffffffff8120c3cb>] dump_header+0x67/0x1d5
Dec 7 11:37:47 vs3 kernel: [312709.328891] [<ffffffff81394f0a>] ? apparmor_capable+0x1aa/0x1b0
Dec 7 11:37:47 vs3 kernel: [312709.328893] [<ffffffff81192ba5>] oom_kill_process+0x205/0x3c0
Dec 7 11:37:47 vs3 kernel: [312709.328896] [<ffffffff811fff2f>] ? mem_cgroup_iter+0x1cf/0x380
Dec 7 11:37:47 vs3 kernel: [312709.328898] [<ffffffff81201ef8>] mem_cgroup_out_of_memory+0x2a8/0x2f0
Dec 7 11:37:47 vs3 kernel: [312709.328899] [<ffffffff81202c97>] mem_cgroup_oom_synchronize+0x347/0x360
Dec 7 11:37:47 vs3 kernel: [312709.328901] [<ffffffff811fdcc0>] ? mem_cgroup_begin_page_stat+0x90/0x90
Dec 7 11:37:47 vs3 kernel: [312709.328902] [<ffffffff811932a4>] pagefault_out_of_memory+0x44/0xc0
Dec 7 11:37:47 vs3 kernel: [312709.328905] [<ffffffff8106af7f>] mm_fault_error+0x7f/0x160
Dec 7 11:37:47 vs3 kernel: [312709.328906] [<ffffffff8106b783>] __do_page_fault+0x3e3/0x410
Dec 7 11:37:47 vs3 kernel: [312709.328909] [<ffffffff81003885>] ? syscall_trace_enter_phase1+0xc5/0x140
Dec 7 11:37:47 vs3 kernel: [312709.328910] [<ffffffff8106b7d2>] do_page_fault+0x22/0x30
Dec 7 11:37:47 vs3 kernel: [312709.328912] [<ffffffff81869038>] page_fault+0x28/0x30
Dec 7 11:37:47 vs3 kernel: [312709.328913] Task in /lxc/1001112/ns killed as a result of limit of /lxc/1001112
Dec 7 11:37:47 vs3 kernel: [312709.328917] memory: usage 1048576kB, limit 1048576kB, failcnt 0
Dec 7 11:37:47 vs3 kernel: [312709.328918] memory+swap: usage 1048576kB, limit 1048576kB, failcnt 59092724
Dec 7 11:37:47 vs3 kernel: [312709.328918] kmem: usage 0kB, limit 9007199254740988kB, failcnt 0
Dec 7 11:37:47 vs3 kernel: [312709.328919] Memory cgroup stats for /lxc/1001112: cache:0KB rss:0KB rss_huge:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
Dec 7 11:37:47 vs3 kernel: [312709.328925] Memory cgroup stats for /lxc/1001112/ns: cache:18168KB rss:1030408KB rss_huge:0KB mapped_file:5052KB dirty:36KB writeback:44KB swap:0KB inactive_anon:524188KB active_anon:524008KB inactive_file:0KB active_file:0KB unevictable:0KB
Dec 7 11:37:47 vs3 kernel: [312709.328932] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Dec 7 11:37:47 vs3 kernel: [312709.329931] [20556] 0 20556 9324 319 22 3 0 0 systemd
Dec 7 11:37:47 vs3 kernel: [312709.329933] [20801] 0 20801 11906 121 27 3 0 0 rpcbind
Dec 7 11:37:47 vs3 kernel: [312709.329935] [20807] 106 20807 45147 129 24 3 0 0 rsyslogd
Dec 7 11:37:47 vs3 kernel: [312709.329937] [20809] 0 20809 6516 66 17 3 0 0 cron
Dec 7 11:37:47 vs3 kernel: [312709.329938] [20851] 0 20851 3210 36 12 3 0 0 agetty
Dec 7 11:37:47 vs3 kernel: [312709.329941] [20852] 0 20852 3210 35 12 3 0 0 agetty
Dec 7 11:37:47 vs3 kernel: [312709.329951] [20854] 0 20854 3210 35 12 3 0 0 agetty
Dec 7 11:37:47 vs3 kernel: [312709.329953] [20899] 0 20899 22428 235 41 3 0 0 saslauthd
Dec 7 11:37:47 vs3 kernel: [312709.329954] [20900] 0 20900 22428 239 41 3 0 0 saslauthd
Dec 7 11:37:47 vs3 kernel: [312709.329957] [24565] 0 24565 255 21 4 2 0 0 AccountingServe
Dec 7 11:37:47 vs3 kernel: [312709.329958] [24995] 0 24995 396041 4390 60 4 0 0 ts3server_linux
Dec 7 11:37:47 vs3 kernel: [312709.330018] [11283] 0 11283 6068 118 16 3 0 0 screen
Dec 7 11:37:47 vs3 kernel: [312709.330020] [11284] 0 11284 1024376 69215 225 7 0 0 java
Dec 7 11:37:47 vs3 kernel: [312709.330036] [24015] 0 24015 8818 1144 20 3 0 0 systemd-journal
Dec 7 11:37:47 vs3 kernel: [312709.330055] [17811] 0 17811 17898 358 39 3 0 0 apache2
Dec 7 11:37:47 vs3 kernel: [312709.330057] [18858] 0 18858 16380 181 35 3 0 -1000 sshd
Dec 7 11:37:47 vs3 kernel: [312709.330058] [18991] 33 18991 139377 479 71 3 0 0 apache2
Dec 7 11:37:47 vs3 kernel: [312709.330060] [19201] 0 19201 3764 48 13 3 0 0 xinetd
Dec 7 11:37:47 vs3 kernel: [312709.330062] [19378] 0 19378 16352 118 22 3 0 0 master
Dec 7 11:37:47 vs3 kernel: [312709.330063] [19380] 108 19380 16881 112 24 3 0 0 qmgr
Dec 7 11:37:47 vs3 kernel: [312709.330065] [20512] 0 20512 25926 163 20 3 0 0 ntpd
Dec 7 11:37:47 vs3 kernel: [312709.330066] [25135] 1001 25135 6069 119 16 3 0 0 screen
Dec 7 11:37:47 vs3 kernel: [312709.330068] [25138] 1001 25138 14130 181 23 3 0 0 sinusbot
Dec 7 11:37:47 vs3 kernel: [312709.330069] [25139] 1001 25139 338314 9646 112 6 0 0 sinusbot
Dec 7 11:37:47 vs3 kernel: [312709.330074] [ 5082] 1001 5082 296249 2205 110 4 0 0 ts3client_linux
Dec 7 11:37:47 vs3 kernel: [312709.330090] [32142] 1001 32142 287213 2344 99 4 0 0 ts3client_linux
Dec 7 11:37:47 vs3 kernel: [312709.330146] [29070] 108 29070 16869 111 25 3 0 0 pickup
Dec 7 11:37:47 vs3 kernel: [312709.330169] [ 904] 0 904 23264 227 48 3 0 0 sshd
Dec 7 11:37:47 vs3 kernel: [312709.330171] [ 2665] 0 2665 4566 128 14 3 0 0 bash
Dec 7 11:37:47 vs3 kernel: [312709.330194] [26574] 0 26574 6069 118 16 3 0 0 screen
Dec 7 11:37:47 vs3 kernel: [312709.330196] [26583] 0 26575 1171519 155512 406 8 0 0 java
Dec 7 11:37:47 vs3 kernel: [312709.330206] [ 5490] 0 5490 6069 76 17 3 0 0 screen
Dec 7 11:37:47 vs3 kernel: [312709.330208] [ 5491] 0 5491 775953 7849 78 6 0 0 java
Dec 7 11:37:47 vs3 kernel: [312709.330209] [ 5541] 0 5541 6069 76 16 3 0 0 screen
Dec 7 11:37:47 vs3 kernel: [312709.330211] [ 5542] 0 5542 762079 2420 68 6 0 0 javaNo users' processes in top, all cores are busy processing kernel tasks - haven't captured what exactly they do, because system was not responding.
Does anyone encountered similar problem? Can't find any info regarding this issue. This problem maybe is not critical at the moment, but it happens from time to time - recently more often. I can't assign more RAM to the container. I would like to avoid hanging and just kill processes utilizing so much RAM (and it happen).
Thanks in advance for any hints!
