Memory creeping

[SiZiGee]

Hi there,

I'm not sure if this is related to specific distros, or its a proxmox thing. But when over time, memory usage creeps up on idling VMs for no reason. After either a VM restart, or host restart, it seems to clear up. But over the course of a couple of weeks, it slowly increases to close to, or over, 80% usage. All VMs are using guest agents, as well.

I will turn on my test VMs (opensuse, centos, ubuntu) again, and post those, as they increase. So far centos (8 stream) was the only one, which showed no memory growth, which perplexed me.

CPU usage was fine and did not increase.

I was hoping that the upgrade from 6 to 7, would clear this issue, but it seems like it didn't. I only have 3 VMs running at the moment (jenkins, pihole and a self written website, which I checked the process memory usage, and stayed stable and low). I am planning on deploying k8s on there (3 more VMs), but the memory growth/creeping is bothering me and would like to choose the right distro, if it is distro specific and not proxmox.

 

[Dunuin]

That shouldn't be an issue. Proxmox will show you the real physical memory usage of that VM. If the guest is using RAM for page file caching the guest will show it to you as "available" (sometimes misinterpreted as "free") but in reality it is just used RAM like everything else. Proxmox just cares if it consumes physical memory (used) or not (free) and won't differentiate on how the RAM is used inside the VM.

Run "free -h" inside your Linux VM and look how much is acutually really "free" and not "available". That "free" should correspond with the "free" your PVE graph is showing. If the "buffer/cache" is growing over time that is normal and that should be the cause of your growing RAM usage.
Look here: https://www.linuxatemyram.com/

 

[SiZiGee]

I'll wait for the memory to start growing, before checking. But it's odd that memory increases over time, except for centos. And the test VMs, have literally nothing on them, besides the guest agent installed and running.

For the historical record of the VMs (just started):

test@opensuse-test-1:~> free -h
total used free shared buff/cache available
Mem: 1.9Gi 101Mi 1.6Gi 16Mi 230Mi 1.7Gi
Swap: 0B 0B 0B

[test@centos-test-1 ~]$ free -h
total used free shared buff/cache available
Mem: 1.8Gi 141Mi 1.4Gi 8.0Mi 194Mi 1.5Gi
Swap: 1.6Gi 0B 1.6Gi

test@ubuntu-test-1:~$ free -h
total used free shared buff/cache available
Mem: 1.9Gi 133Mi 1.3Gi 0.0Ki 508Mi 1.7Gi
Swap: 2.0Gi 0B 2.0Gi

 

[Dunuin]

Your VMs are always creating data. Logs, metrics, ...
These will be cached so the RAM usage should grow.

 

[SiZiGee]

that is true... but that should be written to disk and be cleared in memory though. The memory usage would look similar to centos (was the only distro with no change to memory)

 

[SiZiGee]

Update:
It seems to be distro dependent and not proxmox... well... I have updated proxmox (haven't restarted it yet, no failover nodes yet). I might restart it now, and let it run again and check 1 last time, for science
test centos is still between 15% and 30% (nothing running/installed on it, blank VM)
test opensuse increased to >60% (nothing running/installed on it, blank VM)
test ubuntu increased to ~78% (nothing running/installed on it, blank VM)
windows server (AD) ~54%
ubuntu (single golang web app) ~40% (odd, but fine. still going to migrate over to centos or docker)
fedora (pihole) ~33%