lxc not really "protected" by oom

[parc-network]

Hello fellas,

Recently I switched my magento2 setup, happily only dev and staging environment, from VMs to LXC. So far so good! But I encounter a very annoying problem with the backend servers, where my crons are scheduled. Unfortunately, the lxc hangs/freezes/unresponsible after a random time. Thus it is not usable until I shutdown and start it again.
On my proxmox host

I am really interesssted in a solution to my problem, since adding X amount of ram or cpu, is not really a solution.

I also installed systemd-oomd and added some rules to protect my tasks in general and in addition i added a override-script for my php8.2-fpm instance.

/etc/systemd/oomd.conf

[OOM]
DefaultMemoryPressureLimit=80%
DefaultMemoryPressureDurationSec=10s

If its important: I use pve 8.3.5 on a bare metal installation and my LXC run on ubuntu 22.04 LTS templates.

If I filter on my pve host for the oom's I receive, I get the following output:

journalctl -k | grep -i oom

Apr 02 12:27:29 c-003 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/502,task_memcg=/lxc/502/ns/user.slice/user-0.slice/session-6846.scope,task=php,pid=4084601,uid=101000
Apr 02 12:27:29 c-003 kernel: Memory cgroup out of memory: Killed process 4084601 (php) total-vm:301924kB, anon-rss:176916kB, file-rss:192kB, shmem-rss:0kB, UID:101000 pgtables:552kB oom_score_adj:0
Apr 02 12:27:36 c-003 kernel: php8.2 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr 02 12:27:36 c-003 kernel:  oom_kill_process+0x110/0x240
Apr 02 12:27:36 c-003 kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Apr 02 12:27:36 c-003 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/502,task_memcg=/lxc/502/ns/user.slice/user-0.slice/session-6846.scope,task=php,pid=4084611,uid=101000
Apr 02 12:27:36 c-003 kernel: Memory cgroup out of memory: Killed process 4084611 (php) total-vm:301924kB, anon-rss:176492kB, file-rss:384kB, shmem-rss:0kB, UID:101000 pgtables:552kB oom_score_adj:0
Apr 02 12:27:56 c-003 kernel: php8.2 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Apr 02 12:27:56 c-003 kernel:  oom_kill_process+0x110/0x240
Apr 02 12:27:56 c-003 kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
Apr 02 12:27:56 c-003 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=ns,mems_allowed=0-1,oom_memcg=/lxc/502,task_memcg=/lxc/502/ns/user.slice/user-0.slice/session-6846.scope,task=php,pid=4084603,uid=101000
Apr 02 12:27:56 c-003 kernel: Memory cgroup out of memory: Killed process 4084603 (php) total-vm:301924kB, anon-rss:176240kB, file-rss:576kB, shmem-rss:0kB, UID:101000 pgtables:544kB oom_score_adj:0

my lxc.conf:

arch: amd64
cores: 4
features: nesting=1
hostname: XXX
memory: 4092
mp0: local-lvm:vm-999-disk-1,mp=/mnt/shared,size=102G
nameserver: 8.8.8.8
net0: name=eth0,bridge=dmz,firewall=1,gw=X.X.X.X,hwaddr=BC:24:11:FC:12:F3,ip=X.X.X.X,type=veth
ostype: ubuntu
rootfs: local-lvm:vm-502-disk-0,size=40G
swap: 0
unprivileged: 1
lxc.cgroup2.memory.high: 3500M
lxc.cgroup2.memory.max: 4096M

Additional notes: What leads me to suppose it's somehow related to LXC is that, with the exact same setup (lxc and vm has been configured via my ansible playbooks) I never experienced a freeze/hang/unresponsible state while using a VM.

Thanks for your patience and I am happy for every suggestion!

 

[BobhWasatch]

The OOM-killer is killing a PHP process in the container. You either running low on system memory or your container processes are exceeding the limit you set.

Unlike a VM, the host does not view a container as a single process. It is a collection of normal processes running in a jail. The system calls it makes are handled by the host's kernel. It is not virtualized at all. The host kernel can and will kill individual tasks if necessary.

With a VM the host can't see what processes are inside the VM so it can only kill the whole thing when out of memory. The OS inside the VM can, of course, kill individual processes. But a VM is much more isolated from the host than an container is.

This difference can cause weird issues. For instance, a process might check how much memory is available and allocate some fraction of that. There are various API's to check memory, some of which return host information even inside a container (this is a bug but until it is fixed we are stuck with it). That can't happen in a VM because the OS inside the container simply can't see the host's full memory.

If it worked correctly in a VM maybe you should go back to that setup.

 

[parc-network]

Thank you very much for the detailed explanation — it definitely helps to better understand the underlying differences between LXC containers and traditional VMs.

That said, it unfortunately doesn’t really address my actual issue. While it’s true that I could switch back to a VM, I’m intentionally trying to stick with LXC due to its lightweight nature, faster startup times, and more efficient resource usage. There are plenty of hosting providers offering solutions like Magento 2 inside LXC containers, so this setup clearly can be made stable and production-ready — even with memory-hungry applications like PHP.

I’m therefore looking for a way to properly handle this kind of memory behavior within the LXC context. If anyone has suggestions or best practices — e.g., around memory/cgroup tuning, more precise limits, or ways to prevent the OOM killer from terminating critical processes — I’d really appreciate it.

What I also don’t fully understand is why the container ends up in a frozen state after the OOM killer is triggered. As far as I know, the kernel should terminate a high-memory process (like php-fpm) once a memory threshold is breached, and the system should then recover and remain stable. Instead, the entire container appears to freeze, requiring manual intervention. That behavior seems odd — is this expected, or could it point to a misconfiguration or bug?

 

[BobhWasatch]

Thank you very much for the detailed explanation — it definitely helps to better understand the underlying differences between LXC containers and traditional VMs.

That said, it unfortunately doesn’t really address my actual issue. While it’s true that I could switch back to a VM, I’m intentionally trying to stick with LXC due to its lightweight nature, faster startup times, and more efficient resource usage.

There are theoretical efficiencies, yes. But if it doesn't work efficiency is zero, right? Do you have something to do with a little bit of extra memory? Do you restart the machine a lot? In other words, does this slight gain in efficiency actually matter for your use-case?

There are plenty of hosting providers offering solutions like Magento 2 inside LXC containers, so this setup clearly can be made stable and production-ready — even with memory-hungry applications like PHP.

Do they give only 4 GB of memory? I don't use Magneto so I can't speak to your other questions but from the outside looking in it seems like you are tilting at windmills.

I’m therefore looking for a way to properly handle this kind of memory behavior within the LXC context. If anyone has suggestions or best practices — e.g., around memory/cgroup tuning, more precise limits, or ways to prevent the OOM killer from terminating critical processes — I’d really appreciate it.

This seems like a lot of work for very little gain. Again, it worked in a VM, right?

ETA: I get that homelabbers are all about "learning". But one might consider what you are learning and why and maybe there are more important things to learn about. One of those things might be when it is appropriate to use a container and when it isn't.

 

[BobhWasatch]

What I also don’t fully understand is why the container ends up in a frozen state after the OOM killer is triggered. As far as I know, the kernel should terminate a high-memory process (like php-fpm) once a memory threshold is breached, and the system should then recover and remain stable. Instead, the entire container appears to freeze, requiring manual intervention. That behavior seems odd — is this expected, or could it point to a misconfiguration or bug?

What does "frozen" mean? Can you log in to the container with SSH or pct enter?

In any case, the behavior when something is killed by out of memory is not defined. It depends entirely on what kind of system is in the container, what the dead part was responsible for, etc. As well as on what your definition of "frozen" is.

 

[parc-network]

Well, I’m running around 40 other LXC containers on this specific node — all of which perform reliably under similar conditions. The only containers causing trouble are the two running Magento 2 cron jobs. So yes, in those two specific cases, the efficiency is currently zero — but that doesn’t invalidate the benefits of LXC in general.

That being said: either you have concrete suggestions that could help me identify what’s going wrong, or we’re both just wasting time here

To clarify the issue: when the problem occurs, the container ends up in a frozen state — I can’t access it via pct exec, SSH, or even attach a TTY from the host. Additionally, the CPU usage of the container keeps increasing steadily, even though it’s effectively unresponsive.

My understanding is that the OOM killer is supposed to terminate the process with the highest memory consumption in order to restore stability. In this case, however, something seems to go wrong in that mechanism — the process gets killed, but the container doesn’t recover. That’s the part I’d really like to understand and resolve within the LXC context.

 

[BobhWasatch]

My understanding is that the OOM killer is supposed to terminate the process with the highest memory consumption in order to restore stability.

First, the restoration of stability after the OOM-kill kills something is very much not guaranteed. It is more of a hope. We are out of memory, maybe killing something will help. Often it works. Sometimes things just get worse. I don't know why you think stability will necessarily be restored.

I guess if you really want to investigate you could log in to the container and do some experiments. Manually kill the same process. See what happens, what gets logged, what process is using the CPU, etc. The increasing CPU use sort of says that what got killed was being used by something else that keeps retrying. Perhaps that is eating up all of the CPU resources assigned to the container which prevents starting another shell. Lots of things can happen depending on exactly what's running in there and what the dependencies are.

I think that instead you need to figure out why it is using too much memory and fix that. Adding more RAM or swap to the LXC would probably help. Somehow telling the processes inside to limit their memory use may help.