Memory usage discrepancy

[jaytee129]

Am a couple months into proxmox and have been building an appliance with a Sophos UTM VM (which is the gateway to internet) and a Windows VM. No other VMs but NUT was installed to manage UPS in case that needs to be factored into the problem for some reason.

I allocated 6GB RAM to Sophos VM (needs min 4GB so 6GB plenty) and noticed that after a couple of days of running under very light traffic load, proxmox says it's using over 90% of the allocated RAM to it. What's particularly strange is that Sophos itself is showing well below 50% memory usage internally.

Also, while each VM (Sophos and Windows) separately are showing RAM use of 5.6G and 3.4G respectively for a total of 9 GB, proxmox is showing it's using 13GB of 15.4G total. (There's 16GB RAM on the motherboard). Note, also, that I disabled KSM in case that's relevant.

What's the explanation for this? Is there a memory leak? Is something about go wrong?







Total usage:

 

[leesteken]

Probably filesystem cache, which is not counted as used inside the VM because it can be reused at any time. But the Proxmox host does not know what the memory is used for and counts it a used.
The balloon device can sometimes give better memory usage information if the balloon driver inside the VM supports it (even without actually using ballooning min=max). But then people complain that the kvm process uses more memory than the GUI shows, which is for the same reason as above.
There are lots of related topics in this forum about memory usage, if you want to look into it.

 

[Dunuin]

A VM only sees virtual RAM not real RAM. Also there is always virtualization Overhead, caching and so on. If you really want to know how much physical RAM a VM is consuming install htop and run it on the host to see the KVM processes. "RES" is how much RAM that KVM process virtualizing your VM is really using. Its often way higher than whats the VM is using as virtual RAM.

And for the hosts RAM. In case you are using ZFS its ARC will also use up to 50% of your total RAM for caching.

And VMs will usually eat up all RAM you throw at them. If services/applications doesn't need that RAM it will be used for caching. So its normal that a VM is always at 90+% virtual RAM usage...no matter how much RAM you assign that VM.
If you don't want to waste physical RAM, don't give that VM more RAM as it actually really needs. When you say it only needs max 4GB then I wouldn't give that VM more than 5GB.

 

[jaytee129]

Thanks for the info. For the record I'm using ext4, not ZFS.

It seems proxmox is keeping windows VM RAM down consistently to about 50-60% but never bringing Sophos memory usage down from 90% or so. Is that because the windows VM supports ballooning and Sophos VM does not?

I originally set Sophos at 6GB and Windows at 8GB, leaving 2GB for proxmox overhead because I didn't really need to give the VMs any more than that. When I saw what looked like an over consumption of RAM I brought the Windows VM down to 7 then 6GB but clearly that wasn't necessary. (and it didn't seem right that I had to leave 3-4GB of RAM for Proxmox, which has only these two VMs to manage.)

Can I safely put the Windows VM back up to 8GB or even 9GB? Is 1-2GB of headroom for proxmox plenty, particularly since it seems it can borrow RAM from at least the windows VM, if needed? Can I go to 10GB and flirt with what would seem like overprovisioning territory.

Given my perception (correct me if I'm wrong) that Windows likes to take as much RAM as it can and tunes itself, or at least it's caching, based on available RAM, can I set a minimum that proxmox doesn't ask for unless its critically needed?

I did set both the amount of RAM and the minimum to the same value but that doesn't seem to be preventing proxmox from holding back 40-50% of the RAM from the Windows VM.



Thanks again.

 

[Dunuin]

PVE won't steal any RAM from a VM when min memory is the same as max memeory. If PVE then runs out of RAM it will start swapping making everything super slow or it will even kill the whole VM when you run out of swap. You really should have a look at the processes. Maybe your 6GB WinVm and 6GB Sophos VM are already consuming 7GB+7GB so something similar with nothing left for PVE. If you give a VM 6GB virtual RAM that doesn't mean it will only use up to 6GB of physical RAM. Actual physical RAM usage may be way over that, because of KVM overhead, caching and then you also got stuff like memory fragmentation and so on.

 

[leesteken]

It seems proxmox is keeping windows VM RAM down consistently to about 50-60% but never bringing Sophos memory usage down from 90% or so. Is that because the windows VM supports ballooning and Sophos VM does not?

If Sophos does not support ballooning, Proxmox cannot take memory away. (Unless you set it up for hotplug but that would take manually action to take memory away.)

I originally set Sophos at 6GB and Windows at 8GB, leaving 2GB for proxmox overhead because I didn't really need to give the VMs any more than that

The consensus appears to me that 2G for Proxmox is a good minimum amount, but VM's always have some memory overhead. Swap is useful for memory that is allocated but not actively in use.

Can I safely put the Windows VM back up to 8GB or even 9GB? Is 1-2GB of headroom for proxmox plenty, particularly since it seems it can borrow RAM from at least the windows VM, if needed? Can I go to 10GB and flirt with what would seem like overprovisioning territory.

Ballooning will try to keep host memory (without ext4 filesystem cache) down to 80% according to the manual. The Proxmox host does not really "negotiate" with VMs for memory and it always respects their memory settings. The Proxmox host (with Debian underneath) does normal Linux memory and swap management, and it will OoM-kill large memory-users when it runs out.

Given my perception (correct me if I'm wrong) that Windows likes to take as much RAM as it can and tunes itself, or at least it's caching, based on available RAM, can I set a minimum that proxmox doesn't ask for unless its critically needed?

Linux also tries to use all memory and tune caching based on it. You can set the Minimum memory for each VM with ballooning and ballooning will always leave that amount to the VM.

I did set both the amount of RAM and the minimum to the same value but that doesn't seem to be preventing proxmox from holding back 40-50% of the RAM from the Windows VM.

This disables the VM from ballooning (as it will always leave the Minimum memory to the VM). I don't know how you got to the 40-50%. I assume that Windows just didn't use it yet because it didn't do enough I/O to fill up memory with filesystem cache.