PVE9.x Debian13 LXC /tmp

D

drevilish

Member
Oct 22, 2023
16
3
8
I noticed an issue where I was experiencing a lot of OOM errors within my LXC containers. After updating to Deb13/PVE9.x I found some of my LXC containers at 99% CPU and 99% MEM usage.

Debian 13 has, by default, moved /tmp to a tmpfs.

It incorrectly pulls the amount of memory tmpfs has available as the host memory capacity, in my case 64GB; tmpfs defaults to size=50%. Meaning the LXC thinks it can use 50% of available memory for /tmp.
If I set the container memory limit to 1GB, if /tmp directly tries to store more than 1GB, it's not allowed, and the CPU spikes to 99% and memory usage is stuck at 99%.

"You can return to /tmp being a regular directory by running `systemctl mask tmp.mount` as root and rebooting."

https://www.debian.org/releases/tri...-files-directory-tmp-is-now-stored-in-a-tmpfs
https://manpages.debian.org/trixie/manpages/tmpfs.5.en.html

Please let me know if you think I've missed something.

UPDATE:
A

[SOLVED] Thread 'small LXC + tmpfs + journald with defaults => LXC die with OutOfMemory'

Hello All!

I have been experiencing some LXC with small memory allocation stops, randomly dies silently without any trace of cause. Watching over time, I found that the most probably reason is out of memory. Allocating more memory allows it to run longer, but it fails anyway. When I try to find out where the memory lost, I found that "shared" grows and grows until reach OOM condition.

The result my investigation is very courious, follow the sequence:
1. When container starts, it has tmpfs mount at /run and the size of tmpfs is set to a half of physical server physical memory...
https://bugzilla.proxmox.com/show_bug.cgi?id=6167
Also found an older post where someone was experiencing issues with tmpfs allocating node/host kernel RAM size to the LXC rather than the LXC memory limit.

This issue has caused a number of memory limit problems in my LXC containers.
 
Last edited:
  • Like
Reactions: swdd and leesteken
Indeed, the /tmp has a limit of about half of the host memory (which is 256 times more than the memory setting for this particular CT). Also, systemd-networkd-wait-online was enabled and not working, based on the default Debian 13 CT template.
 
Do you know a way of getting the tmpfs set to limit to the LXC memory limit rather than to the host memory?
 
you need to do that manually per tmpfs instance inside the container. tmpfs has no notion of cgroups unfortunately.
 
  • Like
Reactions: drevilish
Just read all of your comments on the bugzilla see above.
It's crashed out a couple of my LXCs, this means I need to set my mem limits much higher than ideal for the application.
 
For anyone finding this thread later: this is tracked at Bugzilla #6167 (open, UNDECIDED), where I have added measurements across ten containers.

Two things worth adding here.

First, on Fabian's "you need to do that manually per tmpfs instance" - that is right, and the drop-in form works well. In the container:

Code: 
/etc/systemd/system/tmp.mount.d/99-size-cap.conf

[Mount]
Options=mode=1777,strictatime,nosuid,nodev,size=<25% of CT memory>,nr_inodes=1m

The high numeric prefix is deliberate. Drop-ins are read in lexicographic order and Options= in a mount unit is last-wins rather than append, so a cap named 10- would be silently overridden by anything later that also sets Options= - the wrong way round for a safety limit.

Two traps when applying it:
  • systemctl show tmp.mount -p Options reports the live mount rather than the unit's configured options, so it still prints the old size after you write the drop-in and daemon-reload. It cannot verify the change. Use systemctl cat.
  • You cannot remount live inside an unprivileged CT: mount -o remount re-applies the superblock's uid=100000 (the idmap base), which is not a valid uid in the container userns, so it fails with fsconfig() failed: tmpfs: Invalid uid '100000'. Only restarting tmp.mount (which wipes /tmp) or a reboot applies it.

Second, in case anyone is waiting on a kernel fix: it was proposed in 2017 - Yafang Shao, "mm/shmem: set default tmpfs size according to memcg limit" - and rejected by Shakeel Butt, because tmpfs pages are charged to the memcg of the allocator rather than the memcg that performed the mount, and mounting is typically done by a controller outside the job's memcg. Worth capping now rather than waiting.

I have also posted a symptom-titled thread, because this presents as a frozen container with high node load and an idle CPU, and searching those terms does not currently lead anywhere near this discussion: LXC freezes, ssh 'banner exchange' timeout, high load idle CPU

DISCLAIMER: post created with assistance of Claude.
 
Last edited:
You must log in or register to reply here.
Top Bottom
Back