[SOLVED] New Proxmox server has constantly increasing memory usage; is this normal? How to resolve?

[oscaro]

Hi all, will share some details and hope to hear back from you. Thank you in advance for any help.

The issue: RAM usage is constantly increasing even though there are not that many VMs, and the VMs are not using any more RAM than normal.

Server hardware: 16 core AMD Ryzen 5950x, 128GB RAM, 2x 1TB NVME SSD combined in ZFS 1TB pool.

I built a new server and installed Proxmox 7.2-1 on it using ZFS.

I set up a Windows VM and an Ubuntu VM and gave them some CPU and RAM.

Windows VM has 32GB RAM assigned to it, and Ubuntu VM has 8GB RAM assigned.

So I should only be using 40GB of RAM for both the VMs, plus whatever is needed for Proxmox and ZFS cache. Instead, my server is using 70.58% of the current RAM and is always increasing.

Questions I have:

1. Is this normal for this much RAM to be used in this configuration?
2. Is it normal for RAM usage to constantly increase over time?
3. Is KVM supposed to use so much RAM?

Further details:

I think the issue might have something to do with ZFS, because running the arcstat command gives me:

    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  size     c  avail
17:57:46     0     0      0     0    0     0    0     0    0   47G   49G    22G

Free memory command gives me:

              total        used        free      shared  buff/cache   available
Mem:          128747       90051        2770          47       35925       37419
Swap:              0           0           0

When I run top, I get this. See how KVM is using 31.3g / 35.8g of RAM?

top - 17:53:09 up 37 days, 14:06,  2 users,  load average: 0.06, 0.07, 0.08
Tasks: 541 total,   1 running, 540 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.1 us,  0.2 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 128747.2 total,   2760.7 free,  90059.6 used,  35926.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  37411.5 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3368 root      20   0   35.8g  31.3g   5624 S  12.3  24.9  14445:25 kvm
   3604 root      20   0 9288428   5.1g   4952 S   0.7   4.1 361:41.12 kvm
    831 root      20   0       0      0      0 S   0.3   0.0  10:39.85 txg_sync
   3443 root      20   0       0      0      0 S   0.3   0.0 136:23.49 kvm-pit/3368
2374668 root      20   0       0      0      0 I   0.3   0.0   0:05.74 kworker/0:2-events
3495073 www-data  20   0  365728 142892   9168 S   0.3   0.1   0:02.97 pveproxy worker
      1 root      20   0  164424   8420   5276 S   0.0   0.0   0:09.59 systemd

Images
Datacenter summary:


Proxmox Node memory usage over the past year:

Proxmox node memory usage over past month:
Notice how it is constantly creeping up in usage?



Windows VM memory usage over the past year


Ubuntu VM memory usage over the past year

 

[oscaro]

Issue persists and RAM usage continues to climb.

Now RAM usage is at 70.64% (88.82 GiB of 125.73 GiB).

Increase by 0.06% over a day.

Writing it down right now it doesn't seem like a lot.

But I wonder if the server is hitting a "plateau" of RAM usage, or whether it will continue increasing day over day?

Graph of node RAM usage over past week:
Started at 81.49 GiB of of 125.73 GiB


Ended at 88.82 GiB of 125.73 GiB

 

[JSEHV]

Hi @oscaro, maybe this can help you out.

To explain what is happening an what can be done to (if needed), I will show you a host which was just reinstalled and rebooted.
The same can be seen here and the VM's are not using it, but ZFS is. None of the below increases in memory is caused by VM's, but by ZFS ARC.
You can limit the max that ZFS ARC can use, if you want to use the rest of the memory for VM's running in that host.


Currently on a host that has no set ZFS ARC max (so that standard max 50% of mem will be used)
15GB is used by ZFS, 10,5GB is used by VM's running on this host.

Average mem use last hours:



Arcstat:




You can limit the amount of memory that ZFS is using for the ARC.

Set it for the current boot (in this example to 4GB):

echo "$[4 * 1024 * 1024 * 1024]" >/sys/module/zfs/parameters/zfs_arc_max

- Verify the parameter:

cat /sys/module/zfs/parameters/zfs_arc_max

Set it for the next boots (in this example to 4GB (max parameter used form above verification):

nano /etc/modprobe.d/zfs.conf

- Add the following line to the zfs.conf

options zfs zfs_arc_max=4294967296

 

[aaron]

ZFS will use up to 50% of the RAM if it is free for its cache (ARC). It will release the memory if it is needed by other processes.

You can run arcstat to get an idea how much it is using and if that matches what you see.

 

[aaron]

Oh, I just saw that you already checked it.

I think the issue might have something to do with ZFS, because running the arcstat command gives me:

time read miss miss% dmis dm% pmis pm% mmis mm% size c avail
17:57:46 0 0 0 0 0 0 0 0 0 47G 49G 22G

Looks perfectly fine. ZFS is using free RAM to cache data for fast reads.

 

[LnxBil]

used ram is good ram ...if you have free ram, your system has too much. Every OS uses its (really) free ram to cache stuff, even Windows. But Windows does not count cached ram as used, but Linux does. So the discrepancy is that those two worlds use different definitions of free.

 

[oscaro]

Hi @JSEHV, @aaron, and @LnxBil,

Thanks for sharing your insight!

I was not sure if this was expected behaviour, but it's good knowing that this is normal. Now I understand the "c" column in arcstat, which mentions the maximum size of 49G, so there is still a bit more RAM to be used.

I understand now how ARC takes as much unused RAM as possible to take advantage of it, but if RAM is needed for VMs or other purposes, ARC will automatically reduce the amount that it uses. Before I was worried that that RAM was going to be unavailable to future VMs, but I see that is not the case.

I don't think I will limit the amount of RAM ARC has access to because its better to have the RAM in use than just wasted. Also there's no reason to limit the RAM at the moment because I don't want to complicate anything, maybe I will do that in the future if the need arises.

Anyway, thanks a bunch. Marking this thread as SOLVED!

P.S. Here's another post I found that discusses this topic.