High RAM usage (Proxmox, LXC, ZFS)

[zmeister]

Hi guys,

I am just starting out with Proxmox / LXC / ZFS. I have tried Proxmox before and loved it, however this time I have a different setup!

Server specs:
16 GB RAM (DDR4)
2x 450GB NVME SSD (ZFS Mirror)
Intel Xeon E3-1230v6 - 4c/8t - 3.5GHz

I am only using one empty LXC container at the moment - it is not using any RAM (30MB). The node reports that 9GB of RAM is used! And only ~500MB in buffers/cache.

Previously on this same (brand new) node I created another 4-5 containers (they are all stopped now) with variable RAM set - allocated over the 16GB of physical RAM I have, however all of those containers were never used as they were empty and they all reported ~20-30MB RAM used.

The question I have is why an empty node has such high RAM usage. I only have 16GB of RAM, so if ZFS is caching some disk accesses then I don't really need it. I have a very fast NVME SSD drive.

Would you please have a look at the below data to see where this RAM is going? I am not technical enough in this configuration.


root@ns3104805:~# free -m
total used free shared buff/cache available
Mem: 15726 9285 5987 118 453 5996
Swap: 4095 0 4095


Please see the attachment with htop memory screenshot.

I thought this is a bit odd as I would expect the node to only be around 1GB in use. Because buffers/cache is only reporting 500MB, I believe the 8.5GB is active memory in use? This means it will not be freed for the containers to use?

If I only have 6GB of workable memory from 16GB physical RAM, it is very cost inefficient for me with this server.

I would really appreciate your help if you could help me pinpoint where exactly this RAM is being used, and whether any optimisations should be made?

Thank you!
Z

 

[zmeister]

An addition: after reboot the node is reporting 1.2 GB RAM usage.

I will try to replicate the high RAM usage again.

Anyone have any idea of what might be happening though?

Thanks

 

[dietmar]

ZFS uses this as cache. See

https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage

 

[LnxBil]

Anyone have any idea of what might be happening though?

Are you aware of
https://www.linuxatemyram.com/

So in short: Linux caches a lot ... with ZFS, the normal free command is not enough, because as @dietmar pointed out, ZFS does things a little differently. The lesson to learn here is that free RAM is always bad and windows reports all buffers and caches as free, so people believe in this "I need free ram" nonsense.

 

[RobFantini]

I like the info on this thread.As usual great advice from pve support and linux bill.

We have had for awhile the same question/issue as zmeister, so forgive the thread hijack.
---------------------------------------------------------------------------------------------

on most nodes we use rpool + have 10 ceph OSD's .


since pve > datacenter > summary can not be used to determine if a node is low on memory, what would you use to determine that?

AND
with a system having just a few containers and plenty of ram:
pve shows memory usage at 90%

htop shows some swap was used:

Tasks: 804 total,   2 running, 564 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.6 us,  1.8 sy,  0.0 ni, 96.1 id,  0.1 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem : 65855584 total,   635424 free, 56967936 used,  8252224 buff/cache
KiB Swap:  8388604 total,  8332540 free,    56064 used. 49076204 avail Mem

I

# cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=5368709120
options zfs zfs_arc_min=1073741824

no kvm's are used

pce memory config:

 # grep memory /etc/pve/lxc/*.conf
/etc/pve/lxc/100.conf:memory: 1536
/etc/pve/lxc/101.conf:memory: 1024
/etc/pve/lxc/105.conf:memory: 4096
/etc/pve/lxc/131.conf:memory: 6432
/etc/pve/lxc/151.conf:memory: 8192

My question is: How to does one determine that a node is not low on memory?

 

[LnxBil]

My question is: How to does one determine that a node is not low on memory?

Normally, you should monitor swap in/out instead of absolute values. Swapping on your PVE host (or any other server) is always bad. To get better performance while swapping, we often use zram to have a first in-RAM compressed swap layer. You should avoid swapping to disk at all costs. You should also monitor your wait times of your CPU.

Otherwise monitor the free space and buffer/cache. The sum should not be lower than approx. 10% of your system, if you use ZFS, you should also monitor your arc max size. We nowadays use influx and grafana to graph all hosts and find bottlenecks over time. It is important to find and then define your baseline. If you're using CEPH and ZFS, you should have more RAM reserved than using other "simpler" filesystems.

 

[RobFantini]

thanks Bill.

we normally use zram, will recheck that it is installed on all nodes.

I'll look at those tools, we currently use zabbix I'll check if that has something we can use.

also it seems that since Meltdown and Spectre patches to kernel that cpu and memory requirements have increased. the fix for us is to just add ram to pve systems and vm's .

 

[RobFantini]

I checked with 'systemctl status systemd-swap' and zram is working on all systems.


Question: is there a cli tool similar to htop that shows actual free memory available for virtual machines?

 

[LnxBil]

Question: is there a cli tool similar to htop that shows actual free memory available for virtual machines?

You mean free inside of your PVE QEMU hosts?
free shows on your host its available space. This is from a big database system which as actually no free space and that is on purpose, but a lot is available if it is demanded (this machine has also 200GB of hugepages, more on that further down):

$ free -g
              total        used        free      shared  buff/cache   available
Mem:            376         205           1           0         169         168
Swap:            16           0          16

If Linux needs memory, it takes memory and if there is nothing left (free empty and all possible buffers cleared), not recently used parts get swapped out. There are some tweakable settings like vm.swapiness and vm.min_free_kbytes to define when this happens. You can also monitor /proc/buddyinfo for contiguous memory segments. You can also run out of memory, if there are no contiguous segments left, e.g. you require a contiguous 4 MB block, but there is nothing free, your malloc will also fail despite having GBs of free space. Unfortunately, memory fragmentation is a real thing.

One thing I can suggest (only for the sake of completeness) is to improve this a little bit is to use hugepages (2M instead of 4K pages) for your VMs. If you use hugepages, you can monitor used pages more easily, but you have to decide in advance how many GB of RAM go into this. Performance-wise hugepages are in general approx. at least 10% faster for bigger equipped RAM machines due to less overhead in memory maps and fragmentation. Yet I still don't know what the official status on this is. I did some benchmarks with PVE 3.4 back in the days ... but it was a long time ago. The drawback of hugepages is that acclaimed hugepages cannot be used with system application and are reserved or split from the normal memory, so you have some kind of second class memory system, so you can end up with a system that has enough hugepages, but not enough normal pages and you will swap aggressively. This is often a tradeoff between performance and maintainability.

 

[RobFantini]

Bill -

> I You mean free inside of your PVE QEMU hosts?
No on pve host.

'free -g ' is the command i was looking for.

thanks for that and the information on hugepages. I knew 0 about that - will study and implement

PS;
the system that swaped is ok:

# free -g
              total        used        free      shared  buff/cache   available
Mem:             62          54           0           0           7          46
Swap:             7           0           7

 

[LnxBil]

Yes, looks ok and you have roughly 60% available.