[SOLVED] LXC showing shared memory from host

[ca_maer]

Hello,

It's seems the shared memory from an LXC container shows the shared memory from the host.

Here's the host:

             total       used       free     shared    buffers     cached
Mem:         64414      62599       1815       1833       9329      31029
-/+ buffers/cache:      22241      42173
Swap:         8191          0       8191

And now an LXC container (Ubuntu 16.04) on the host:

              total        used        free      shared  buff/cache   available
Mem:          10240        1914        1038        1833        7286        1038
Swap:             0           0           0

This cause issue by showing invalid ram usage in software that monitors ram usage such as htop or Munin.

Thanks

proxmox-ve: 5.1-34 (running kernel: 4.13.8-2-pve)
pve-manager: 5.1-41 (running version: 5.1-41/0b958203)
pve-kernel-4.13.4-1-pve: 4.13.4-26
pve-kernel-4.13.8-2-pve: 4.13.8-28
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.13-3-pve: 4.13.13-34
pve-kernel-4.13.13-1-pve: 4.13.13-31
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-18
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-15
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9
libpve-apiclient-perl: 2.0-2

 

[Alwin]

This depends on the tool in the container, you can see it anyway with the sysinfo call.
http://man7.org/linux/man-pages/man2/sysinfo.2.html

 

[ca_maer]

This depends on the tool in the container, you can see it anyway with the sysinfo call.
http://man7.org/linux/man-pages/man2/sysinfo.2.html

I'm not sure I understand correctly. By tool you mean services running inside the container ? The issue is the same across all containers on the same host. I mean the shared memory will be the same on Container A,B and C if they are located on the same host.

How I'm I supposed to use the sysinfo call ? I've tried creating a script with some examples found online but to no avail.

Here's the host:

total        used        free      shared  buff/cache   available
Mem:         128934       60183        4168        3216       64582       64465
Swap:          8191           0        8191

Container A:

total        used        free      shared  buff/cache   available
Mem:            512          59         385        3216          67         385
Swap:             0           0           0

Container B:

total        used        free      shared  buff/cache   available
Mem:           2048         102        1861        3216          84        1861
Swap:             0           0           0

Thanks

 

[Alwin]

It greatly depends on the tool, how it is getting the information. The tool 'free' might just use the sysinfo call and get stats not specific to the namespace of the container.

lxc-cgroup -n <VMID> memory.stat
OR
cat /sys/fs/cgroup/memory/lxc/<VMID>/memory.stat
OR
lxc-info -n <VMID> -S

<VMID>, has to be changed.

Some people already made scripts for checking LXC:
https://github.com/Napsty/check_lxc
https://github.com/munin-monitoring/contrib/tree/master/plugins/lxc

Basically the point is, don't monitor from inside a container, do it on the host.

 

[ca_maer]

Hey Alwin,

Yes I'm already using lxc-info to monitor my container ram usage. After some research if found this bug on the lxcfs github that was resolved 2 months ago and it seems exactly like my issue.

The issue is the following:

MemAvailable is somehow reporting the same value as MemFree in lxcfs. It seems like Cached is not excluded from MemAvailable.

Here's the thread: https://github.com/lxc/lxcfs/issues/175
Here's the commit to fix the issue: https://github.com/lxc/lxcfs/commit...9d39834#diff-0ec86d1df37abb99d3a8ac52b6a749ac

As of now, if a container has cache there is now way to properly calculate the real memory usage with the available tools.

Anyway you guys can include this fix so that tool like lxc-info report correct ram usage instead of 100% because the cached ram is calculated as used.

Thanks

 

[Alwin]

I agree, this commit, should be included in our lxc-pve package, but this commit will not take care of the shared memory displayed.

 

[ca_maer]

I agree, this commit, should be included in our lxc-pve package, but this commit will not take care of the shared memory displayed.

Right but at least it will prevent tools like lxc-info to report 100% ram usage when in fact it is not which is a big deal for people using such tools to monitor ram usage from production systems like us.

Like you said, we should monitor the ram usage from the host using the built in tools and not in the container using htop and free so the shared memory is not really that important. I created the thread because I thought the shared memory was the issue when in fact it was the available memory calculation that was wrong.

Hopefully this could be fix quickly as it seems like a simple fix.

Thanks

 

[Alwin]

Add this to our bug tracker for reference.
https://bugzilla.proxmox.com/show_bug.cgi?id=1655

 

[ca_maer]

Add this to our bug tracker for reference.
https://bugzilla.proxmox.com/show_bug.cgi?id=1655

Thanks Alwin. I will monitor the bug report closely. Cheers

 

[ca_maer]

Hey Alwin,

Just wanted to let you know I've installed the updated LXCFS package and everything is ok now ! Thanks for your help