PVE node RAM consumption

[milanv]

Hey I have a home server Virtual Environment 8.3.2 running few containers and VMs. I have noticed from node summary Im running out of memory. (29/32 GB used)


Though I cant identify why as my VMs are consuming just a few gigs, ZFS arc chache is on 4 gigs.

Running

ps aux --sort -rss

on my node console show VM 104 and 101 are using 25% -> 8GB


The VMs summary show something different though.


sceen from VM101 free -h


Where is the issue then?

 

[chrcoluk]

You only showed the summary for 2 of the 4 VM's, and they may not need to fully utilise their ram, the ram might still be allocated from the host.
In addition is page cache for host OS and ZFS dirty cache (both not part of ARC).
What is free command showing for the host?

 

[milanv]

free command for host

As per ps aux --sort -rss it obviously thinks VM 101 and VM 104 are consuming 8GB. The VM summary show different value. Screen from VM 101 as example show also different values.

 

[leesteken]

As per ps aux --sort -rss it obviously thinks VM 101 and VM 104 are consuming 8GB.

Both are taking 8GB because you set the VM memory to 8GB. This is normal and Proxmox does not support over-commiting on memory.

The VM summary show different value. Screen from VM 101 as example show also different values.

The QEMU Guest Agent and the balloon driver (if installed on the VM) try to show you what the operating system inside the VM shows (which can pretend that filesystem cache is free).

Maybe Proxmox should document this better and explain more about cache and how the VM can reuse cache but Proxmox cannot reuse the memory of the VM. There are two ways of looking at cache memory (free versus used) and people want the first even though for the Proxmox host it's the second. Windows people want to keep memory free, Unix people want memory to be used (for cache) instead of wasted. There is a new thread about this every week.

 

[chrcoluk]

I dont know how the per VM stats are calculated, but I expect it is, it considers zeroed memory (not allocated since boot) as unused, but on the host that memory is still allocated to the VM, so I would say the host is the one to go by.

I see @leesteken just confirmed this, so yeah thats whats going on.

 

[milanv]

First of all its super confusing - Proxmox shows in its GUI - VM summary - its consuming only 1GB. But on other tab - node summary it shows it consumes 8GB. Two tabs, same items and shows inconsistent values.

 

[leesteken]

First of all its super confusing - Proxmox shows in its GUI - VM summary - its consuming only 1GB. But on other tab - node summary it shows it consumes 8GB. Two tabs, same items and shows inconsistent values.

Different people have different ideas of what "memory in use" means and what they want to be shown in the GUI.
You can simplify it to "for Proxmox the VM always uses all of its memory" (and some more because of the overhead of virtualization), regardless of that the GUI shows.

 

[milanv]

Different people have different ideas of what "memory in use" means and what they want to be shown in the GUI.
You can simplify it to "for Proxmox the VM always uses all of its memory" (and some more because of the overhead of virtualization), regardless of that the GUI shows.

There can be different approaches on what "memory in use" means, but you should not dont mix them all together in one metric.

 

[milanv]

Node Search tab:

ps aux --sort -rss


Host Memory usage % - This cant be more obvious flaw.

 

[chrcoluk]

Look at it like this, the summary tab for the entire node, the one with the graphs, is the actual memory allocated. That is the one I go by.
The graph for the VM itself, is showing you how the VM is using the memory assigned to it.
My node search has disk usage, memory usage, cpu usage, uptime, host cpu usage, and host memory usage in that order, all of them except uptime are as percentage. My figures look accurate so e.g. TrueNAS is using almost all of the 8 gig allocated to it, memory usage is correctly reported at 92%, and host memory usage is reported as 12% which is correct as the host has 64 gigs of ram.

Ballooning might help you potentially. https://pve.proxmox.com/wiki/Dynamic_Memory_Management

I might raise a report suggesting they change memory usage and memory usage % columns so it shows the allocated resource, as thats how the the cpu usage column works.

So e.g. "92% of 8 GiB" instead of "92%"

 

[milanv]

Look at it like this, the summary tab for the entire node, the one with the graphs, is the actual memory allocated. That is the one I go by.
The graph for the VM itself, is showing you how the VM is using the memory assigned to it.
Node search looks like its just an easy way to show the per VM summaries, I agree that column header should be renamed to something else like "Guest memory usage".

Ballooning might help you potentially. https://pve.proxmox.com/wiki/Dynamic_Memory_Management

There already is a guest memory usage. It is called "Memory usage %". I have ballooning set on. I think the issue might be im passing through pcie device (iGPU to VM 101) which would explain, Balllooning is disabled and dynamical RAM allocation doesnt work. But the gui graph should reflect this setting. Proxmox know i have PCI device connected. It should report full ram utilization on host then and gray out balllooning option.

 

[chrcoluk]

There already is a guest memory usage. It is called "Memory usage %". I have ballooning set on. I think the issue might be im passing through pcie device (iGPU to VM 101) which would explain, Balllooning is disabled and dynamical RAM allocation doesnt work. But the gui graph should reflect this setting. Proxmox know i have PCI device connected. It should report full ram utilization on host then and gray out balllooning option.

"Memory usage %" and "Memory usage" are both for the guest, the host is "Host memory %", there doesnt seem to be a byte value for the host memory usage, so that is % only.

 

[milanv]

What i dont understand from the Documentation https://pve.proxmox.com/wiki/Dynamic_Memory_Management if i have PCI device pass through set on one VM, does it disable Ballooning for all other VMs?

 

[chrcoluk]

My interpretation is its just the guest with the pass through that cant use ballooning, but thats just my opinion.

 

[leesteken]

if i have PCI device pass through set on one VM, does it disable Ballooning for all other VMs?

No, ballooning does not work only for the VM with PCI(e) passthrough because PCI(e) devices have Direct Memory Access (DMA) and therefore all VM memory must be pinned/locked into actual host RAM (for that specific VM).

 

[BobhWasatch]

One sure way to get a VM to use less RAM is to allocate less RAM to said VM. I really don't understand "I gave it X GB and something must be wrong because it uses X GB". Really odd. If you didn't want it to use the RAM why did you allocate it?

 

[milanv]

One sure way to get a VM to use less RAM is to allocate less RAM to said VM. I really don't understand "I gave it X GB and something must be wrong because it uses X GB". Really odd. If you didn't want it to use the RAM why did you allocate it?

Ahhh. Are just trolling or you really dont understand principle of shared resources?

 

[BobhWasatch]

Ahhh. Are just trolling or you really dont understand principle of shared resources?

RAM is not shared between virtual machines.

 

[milanv]

RAM is not shared between virtual machines.

Oh and whats KSM then? Please if you have nothing helpful to say to this topic, please come another time...

 

[BobhWasatch]

To elaborate a little;

Linux RAM allocation is "lazy". When you allocate a chunk of memory it is not really committed until something touches it. Until then it is just an entry in the page table. In some cases you do need to have that memory be committed and in RAM rather than, say, in swap. There is a system call to do that separate from the ones that allocate the memory.

So when your VM starts up it allocates however many GB you allocated to it. But it doesn't use it all right away. So there's a bunch that is allocated but not committed. This gives the illusion that there is more free RAM than there actually is, when looking from the hypervisor side. Eventually most VM's will end up using most of their RAM and it will all be committed. If you have over-committed RAM, this is where the balloon driver steps in and takes away some RAM from one or more VM's. Or the OOM-killer kills one of them.

VM's that have PCI devices passed through use the allocate-and-lock system call I mentioned above because you can't DMA to a not-present page.

KSM is a special case where kernel pages that have the same content can be shared. Most pages aren't like that. And as soon as one of the shared group is written, it can no longer be shared. It may eventually get combined with some other pages that have the same content. Most KSM pages are going to be code pages rather than data pages.