Lxc container does not free up cached memory

rofrofrof1

New Member
Jan 6, 2025
6
2
3
With some of our Lxc containers, we have observed that ct's cached memory is constantly increasing. As soon as almost the entire memory is occupied, the CPU utilisation in the container sporadically rises to 100% and the container is no longer accessible. Restarting the container (and thus clearing the cache) solves the problem.

I was also able to create the problem artificially by lowering the soft memory limit of the container during operation with:

echo 5113695436 > /sys/fs/cgroup/lxc/123/memory.high
echo 5113695436 > /sys/fs/cgroup/lxc/123/ns/memory.high

Here, too, the CPU within the container increased to 100%, which is why I assume that this is the usual behaviour when the memory of a ct is full. However, most of the memory in the container is only occupied with cached memory, which should actually be released automatically?

The hypervisor does not have a swap partition as we use Zfs and did not want to use swap. Could there be a connection with this?

Do you have any ideas what else I could try?

sc1.png
sc2.png

# cat /etc/pve/lxc/123.conf
arch: amd64
cpulimit: 4
features: mknod=1,nesting=1
hostname: 123.xyz.com
memory: 6144
mp0: freenas:123/vm-123-disk-0.raw,mp=/backups,size=1000G
nameserver: 127.0.0.1
net0: name=eth0,bridge=vmbr0,firewall=1,gw=111.111.111.111,gw6=xxxx:xxxx:xxxx::x,hwaddr=BC:24:11:DA:CD:64,ip=111.111.111.111/24,ip6=xxxx:xxxx:xxxx/56,type=veth
onboot: 1
ostype: centos
rootfs: local-zfs2:subvol-123-disk-0,size=100G
swap: 512
unprivileged: 1

# pveversion -v
proxmox-ve: 8.4.0 (running kernel: 6.8.12-7-pve)
pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)
proxmox-kernel-helper: 8.1.1
proxmox-kernel-6.8: 6.8.12-9
proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9
proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8
proxmox-kernel-6.8.12-7-pve-signed: 6.8.12-7
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ceph-fuse: 17.2.7-pve3
corosync: 3.1.9-pve1
criu: 3.17.1-2+deb12u1
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx11
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-5
libknet1: 1.30-pve2
libproxmox-acme-perl: 1.6.0
libproxmox-backup-qemu0: 1.5.1
libproxmox-rs-perl: 0.3.5
libpve-access-control: 8.2.2
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.1.0
libpve-cluster-perl: 8.1.0
libpve-common-perl: 8.3.1
libpve-guest-common-perl: 5.2.2
libpve-http-server-perl: 5.2.2
libpve-network-perl: 0.11.2
libpve-rs-perl: 0.9.4
libpve-storage-perl: 8.3.6
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.6.0-2
proxmox-backup-client: 3.4.0-1
proxmox-backup-file-restore: 3.4.0-1
proxmox-firewall: 0.7.1
proxmox-kernel-helper: 8.1.1
proxmox-mail-forward: 0.3.2
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.10
pve-cluster: 8.1.0
pve-container: 5.2.6
pve-docs: 8.4.0
pve-edk2-firmware: 4.2025.02-3
pve-esxi-import-tools: 0.7.3
pve-firewall: 5.1.1
pve-firmware: 3.15-3
pve-ha-manager: 4.0.7
pve-i18n: 3.4.2
pve-qemu-kvm: 9.2.0-5
pve-xtermjs: 5.5.0-2
qemu-server: 8.3.12
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.7-pve2
 
That the RAM is used for caching is as expected and not a problem see https://www.linuxatemyram.com/ for reference.

What puzzles me that your container is not accesible due to 100% RAM usage, this sounds like a memory leak in one of the applications you run in the container. What workloads/applications/services are you running inside the lxc?
 
My point is not that unused memory is used for caching, but that it is apparently not freed up when it is needed.

I use Plesk in the containers. I.e. more precisely: apache, smtp, imap-pop3, pop3, pop3s, imap, imaps, authdaemon, dns, postgresql, spamassassin, milter, nginx, fail2ban, resctrl, plesk-php81-fpm, plesk-php83-fpm, plesk-php82-fpm

And It is probably not due to the php web application, as it occurs on several containers that host different websites.
 
  • Like
Reactions: f.sennj
Hello i face the sambe problem i guess, container starts all fine and then after some hours or days it starts like going this:
1747807865620.png

its a simple webserver with apache and php.

after restart it looks again like this:
1747807889748.png

proxmox-ve: 8.4.0 (running kernel: 6.8.12-9-pve)pve-manager: 8.4.1 (running version: 8.4.1/2a5fa54a8503f96d)proxmox-kernel-helper: 8.1.1proxmox-kernel-6.8.12-10-pve-signed: 6.8.12-10proxmox-kernel-6.8: 6.8.12-10proxmox-kernel-6.8.12-9-pve-signed: 6.8.12-9proxmox-kernel-6.8.12-8-pve-signed: 6.8.12-8proxmox-kernel-6.8.12-6-pve-signed: 6.8.12-6proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2ceph: 19.2.1-pve3ceph-fuse: 19.2.1-pve3corosync: 3.1.9-pve1criu: 3.17.1-2+deb12u1glusterfs-client: 10.3-5ifupdown2: 3.2.0-1+pmx11ksm-control-daemon: 1.5-1libjs-extjs: 7.0.0-5libknet1: 1.30-pve2libproxmox-acme-perl: 1.6.0libproxmox-backup-qemu0: 1.5.1libproxmox-rs-perl: 0.3.5libpve-access-control: 8.2.2libpve-apiclient-perl: 3.3.2libpve-cluster-api-perl: 8.1.0libpve-cluster-perl: 8.1.0libpve-common-perl: 8.3.1libpve-guest-common-perl: 5.2.2libpve-http-server-perl: 5.2.2libpve-network-perl: 0.11.2libpve-rs-perl: 0.9.4libpve-storage-perl: 8.3.6libspice-server1: 0.15.1-1lvm2: 2.03.16-2lxc-pve: 6.0.0-1lxcfs: 6.0.0-pve2novnc-pve: 1.6.0-2proxmox-backup-client: 3.4.1-1proxmox-backup-file-restore: 3.4.1-1proxmox-firewall: 0.7.1proxmox-kernel-helper: 8.1.1proxmox-mail-forward: 0.3.2proxmox-mini-journalreader: 1.4.0proxmox-offline-mirror-helper: 0.6.7proxmox-widget-toolkit: 4.3.10pve-cluster: 8.1.0pve-container: 5.2.6pve-docs: 8.4.0pve-edk2-firmware: 4.2025.02-3pve-esxi-import-tools: 0.7.4pve-firewall: 5.1.1pve-firmware: 3.15-3pve-ha-manager: 4.0.7pve-i18n: 3.4.2pve-qemu-kvm: 9.2.0-5pve-xtermjs: 5.5.0-2qemu-server: 8.3.12smartmontools: 7.3-pve1spiceterm: 3.3.0swtpm: 0.8.0+pve1vncterm: 1.8.0zfsutils-linux: 2.2.7-pve2
 
What does free -h and top -co %MEM inside the CT look like when it is in this state?
 
Last edited:
problem is i cant even log into the CT unless i restarted or increased the ram. so i cant give output. i guess the thing in common is that PHP and apaches is installed. is there a probleme with mem over commit in php?
 
I've been testing out LXC and getting ready to roll out for production but I ran into the exact same issues with LXCs and had been pulling my hair out to figure out why they were dropping like flies (but appeared to be working). I couldn't even ssh in and open a terminal on them once it hits a reasonable load. Until there's a load it seemed fine. Going to keep a watch on this ...maybe LXCs are not a good choice for me right now! (which is a bit scary as I based all of my platform design on them!)
 
problem is i cant even log into the CT unless i restarted or increased the ram. so i cant give output. i guess the thing in common is that PHP and apaches is installed. is there a probleme with mem over commit in php?

I'm not running PHP at all on my LXCs that have the same issue. Running Python based ERP (Odoo) and have the same thing going on. When it hits high resources I can no longer even log in either with console or ssh and the LXC stops serving content (but the CPU and memory graphs stay very high and moving)