How to make reported LXC memory usage include its cache/buffer?

[YungErrorHunter]

The reported memory usage of a VM in its summary and graphs includes RAM used by caching inside a VM.

The reported memory usage of a node in its summary includes RAM used by caching inside VMs and LXCs.

The reported memory usage of an LXC in its summary does not include RAM used by caching.

Is this some technical limitation, an intended feature, a bug, or simply something I can easily change? I would prefer if the memory usage reported for an LXC would behave like it does with VMs, so it's easy for me to see how much actual RAM is taken from the host by a given LXC.

(In case it matters: postgresql running inside an LXC is using 16GB of RAM for caching, however the GUI reports that the LXC is only using 2GB of RAM at the moment. Those same 16GB are however included, as it should imo, in the node's RAM usage.)

From inside the LXC:

~# free -h
               total        used        free      shared  buff/cache   available
Mem:            32Gi        17Gi        14Gi        16Gi        16Gi        14Gi
Swap:             0B          0B          0B

 

[YungErrorHunter]

I found this thread: https://forum.proxmox.com/threads/wrong-memory-usage-reported-for-lxc-running-vms.158035/ which, if I interpret it correctly, suggests that proxmox should be using the value from /sys/fs/cgroup/lxc/119/memory.current for LXC memory usage. (119 is my LXC id)

cat /sys/fs/cgroup/lxc/119/memory.current on the host (and cat /sys/fs/cgroup/memory.current within the LXC) both report almost 18GB, which is the amount of host memory the LXC is actually using at the moment (sum of used+cached) and which I'd want to be displayed on the LXC's summary and in its graphs, but somehow that is not the case.

Instead, it seems to be using the kernel value from /sys/fs/cgroup/lxc/119/memory.stat. I don't know if that's intended or not. Is there anything I can do to change this?

 

[YungErrorHunter]

So I found https://github.com/proxmox/pve-manager/blob/master/PVE/PullMetric.pm - VMs here have "mem" and "memhost" , whereas LXCs only seem to have "mem", as reflected by the GUI. (info: https://git.proxmox.com/?p=pve-manager.git;a=commit;h=e4ca137de81350cf4df79ec956dde568f00ff812)

Further down the rabbit hole, finding this: https://git.proxmox.com/?p=pve-mana...3;hb=24ccdc29b34ea498d5a63f8848b608319948e73b it gets LXC infos from PVE::LXC::vmstatus(); which is defined here: https://git.proxmox.com/?p=pve-cont...4a0db19c7ef630d6059e26bff774eb6429f62;hb=HEAD

In there, it seems that LXC memory usage is taken via PVE::LXC::CGroup->new($vmid)->get_memory_stat();

AND FINALLY.... This is the file of interest: https://git.proxmox.com/?p=pve-cont...8;hb=8b882cd4b2c57848207c49c164ad547cdfa40778

For cgroups2, it gets memory usage from /sys/fs/cgroup/lxc/119/memory.current - This is how I'd expect it to work. However, in my case, this would be (as of this post) 22.3GB. This is the value I want in the GUI. But when I look at the GUI, the value is 2.08GB instead (the value from memory.stat->kernel).

Looking at /run/pve/metrics, the file where the GUI gets its data from, "mem" reports "2452283392". Where does it get this from?!

I'm at a dead end. I can't find the file responsible for populating /run/pve/metrics. Any help?

 

[YungErrorHunter]

OK found it. The file is https://git.proxmox.com/?p=pve-comm...55be490775caeeb4b09e8b67;hb=refs/heads/master

This is the relevant part of the code:

 350         my $mem = file_get_contents("$path/memory.current");
 351         my $swap = file_get_contents("$path/memory.swap.current");
 352         my $stat = parse_flat_keyed_file(file_get_contents("$path/memory.stat"));
 353
 354         chomp($mem, $swap);
 355
 356         $res->{mem} = $mem - $stat->{file};
 357         $res->{swap} = $swap;

Why are we subtracting $stat->{file} here? If we would simply return $mem we would get the correct value of memory used including cached memory, in other words: The amount of RAM the LXC is actually taking away from our host's memory. Surely this is what we'd want right? Right?...

Or, couldn't we just report both values and populate "mem" and "memhost" for metrics/stats/graphs like we do with VMs?

 

[YungErrorHunter]

Bump. This behaviour is really annoying. Especially since the "Host memory usage" on the Datacenter -> Search overview is also wrong, making it hard to easily see how resources of a node are allocated. Proxmox API is of course also reporting the wrong value only, "memhost" like it exists for VMs does not exist for LXCs.

I refuse to believe that this could be intended behaviour at all, especially when it's done different (right) for VMs.

 

[cheiss]

Hi,

bug- and feature reports should go on our bugtracker: https://bugzilla.proxmox.com/
It allows to properly track reports and things easily get lost in the forum here
Also, please link this topic.

The amount of RAM the LXC is actually taking away from our host's memory. Surely this is what we'd want right? Right?...

Caching is a hugely complex topic. As LXCs share the kernel with the host system, (file system) caches might be shared - not too familiar with how exactly cgroups v2 does memory accounting though.

Showing the cache similar to host/VM might sense - but this is probably best decided on the bugtracker.