[LXC Stability] Cached memory leaks (killing processes (wtf!))

[ca_maer]

re: vi /etc/systemd/journald.conf ,

we are starting out with these settings. could you tell me what changes you've made ?

RuntimeMaxFileSize=50
RuntimeMaxFiles=100

PS: solving the lxc memory thing is huge.

It really depends on what your container can afford and the logging you want to keep.

For example, if your container as 2GB ram and you allocate 3GB to log journaling then the tmpfs will get full and the OOM will be called as the host will think the container use 3GB of ram of the 2GB allocated.

Since we have some containers with some low memory we use the following settings for all our containers in /etc/systemd/journald.conf

RuntimeMaxFileSize=5M
RuntimeMaxFiles=3

YMMV so tweak those according to your need

 

[RobFantini]

ca_maer thanks for the advice

also I have noticed since our desktop systems upgraded from jessie to stretch - a lot more OOM issues. so this is not a proxmox only issue for us. we've added memory to about 5 desktops to solve oom .

 

[decibel83]

Hi, I'm suffering this problem on two different Proxmox hosts with two LXC containers:

[338962.945187] Memory cgroup out of memory: Kill process 33823 (celery) score 6 or sacrifice child
[338962.946422] Killed process 33823 (celery) total-vm:212304kB, anon-rss:51236kB, file-rss:28kB, shmem-rss:0kB
[338973.317470] memory: usage 8388608kB, limit 8388608kB, failcnt 16773127
[338973.317471] memory+swap: usage 8388608kB, limit 29360128kB, failcnt 0
[338973.317472] kmem: usage 77912kB, limit 9007199254740988kB, failcnt 0
[338973.317473] Memory cgroup stats for /lxc/8205: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[338973.317492] Memory cgroup stats for /lxc/8205/ns: cache:8027528KB rss:283168KB rss_huge:0KB shmem:8026296KB mapped_file:4136KB dirty:0KB writeback:0KB swap:0KB inactive_anon:7929340KB active_anon:380124KB inactive_file:712KB active_file:380KB unevictable:0KB

I have this error repeatedly each day and the only solution is to restart the container.

Today I tried to edit the journald configuration before restarting the container but this is the result:

/etc/systemd/journald.conf:

RuntimeMaxFileSize=10M
RuntimeMaxFiles=5

After editing the journald configuration file:

root@processor1:/home/mattia# journalctl --vacuum-size=50M
Vacuuming done, freed 0B of archived journals from /run/log/journal/d91e9335395744a4975d0871068ebec8.
root@processor1:/home/mattia# systemctl kill --kill-who=main --signal=SIGUSR2 systemd-journald.service
root@processor1:/home/mattia# journalctl --vacuum-size=50M
Vacuuming done, freed 0B of archived journals from /run/log/journal/d91e9335395744a4975d0871068ebec8.
root@processor1:/home/mattia# free
              total        used        free      shared  buff/cache   available
Mem:        8388608      387252     4012500       24800     3988856     8001356
Swap:      29360128           0    29360128

top:

top - 20:54:42 up 1 day, 22:53,  1 user,  load average: 0.67, 0.68, 0.72
Tasks:  29 total,   2 running,  27 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.3 us,  0.0 sy,  0.0 ni, 98.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8388608 total,  3990696 free,   409044 used,  3988868 buff/cache
KiB Swap: 29360128 total, 29360128 free,        0 used.  7979564 avail Mem

htop:


Without restarting the container the celery process does not start at all.

Could you help me to understand where I am wrong, please?

 

[flotho]

use persistent logging in LXC,
https://forum.proxmox.com/threads/o...emory-cgroup-out-of-memory.65106/#post-336131