RAM Display Bug on PVE 9.0.3 ?

[Tchangalo]

I just installed PVE 9.0.3 fresh for the first time and set up a pfSense. After the installation, according to the display in the PVE Summary, the 1.5 GiB of RAM assigned to the pfSense was about 102% full. If I increase the pfSense’s RAM to 16 GiB, the PVE Summary shows the RAM usage at 96.79% full, while pfSense itself reports only 3% usage. This happens both with and without ballooning enabled, and also occurs both with and without the Guest Agent installed and enabled in the pfSense.

As a check, I performed exactly the same steps on PVE 8.4.6. In that case, this phenomenon did not occur.

 

[UdoB]

about 102% full.

If I remember correctly it takes into account now that not only the configured RAM but also some overhead on the host is used. With "configured=100%" you land above that value...

I am considering this "normal"

 

[fiona]

Hi,
please see: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#VM_Memory_Consumption_Shown_is_Higher

 

[Tchangalo]

Thank you very much for your quick answer. I have two follow up questions about this part:

"If the new Host memory usage field is the same as the Memory Usage field in the VM's summary panel, then Proxmox VE was unable to gather the VM's internal memory usage information.

This happens if one of the following conditions applies:
[...]
For example, FreeBSD is known to not report memory usage details, which includes popular firewalls like pfSense or OPNsense."

1) As far as I understand this explanation, the RAM usage of the pfSense on the PVE display should actually not reflect the host’s view (due to the limited accuracy of RAM usage reporting on BSD). However, in my case, it does.

2) Where does the host’s view of RAM usage come from, and what does it mean?

 

[aaron]

1) As far as I understand this explanation, the RAM usage of the pfSense on the PVE display should actually not reflect the host’s view (due to the limited accuracy of RAM usage reporting on BSD). However, in my case, it does.

"If the new Host memory usage field is the same as the Memory Usage field in the VM's summary panel, then Proxmox VE was unable to gather the VM's internal memory usage information.

This is regarding the VM summary panel on Proxmox VE. In your screenshots you have "Memory Usage" with the line chart and one line below that is the "Host memory usage". Those values are the same in both screenshots → Proxmox VE did not get any infos from the guest itself and therefore shows how much memory the VM is using on the host itself.

pfSense itself shows 3% used out of the 16 GiB, which is somewhere around 1/2 GiB.

2) Where does the host’s view of RAM usage come from, and what does it mean?

Maybe I elaborate a bit more. To get the guests view, Proxmox VE is using the "ballooning device" to communicate with the VM. You can check it yourself if you go to the "Monitor" panel of a VM and enter info balloon. Compare the reported infos with the pfSense VM and another VM. If reported in detail, there should be "total_mem" and "used_mem". The difference is what is shown as "used".

If those values are not reported back, the "Memory Usage" filed shown in the web UI will have the same value as the hosts view. Prior to Proxmox VE 9, gathering the hosts view was not very accurate and would report a lower value as used. But that was still very different than from what the VM itself would see as used.

With Proxmox VE 9, the hosts memory view should be very accurate now, as for each process in the cgroup of a VM is checked for its PSS memory usage. There can be multiple processes belonging to a VM, as besides the main KVM process, you might have one providing the TPM functionality, for example.


I hope this clears up some confusion that came with this.

 

[Tchangalo]

Yes, thats very helpfull. But for me there's one mystery left: Why does the PSS memory usage covary with the absolute RAM size? See pics.

My intuition tells me, that the RAM usage value is located in a fixed spectre between IDLE and hard work, but this spectre as such will not move up the scale, so to say, when RAM size gets larger? Or lets put it this way: why does it?

 

[aaron]

Why does the PSS memory usage covary with the absolute RAM size? See pics.

My intuition tells me, that the RAM usage value is located in a fixed spectre between IDLE and hard work, but this spectre as such will not move up the scale, so to say, when RAM size gets larger? Or lets put it this way: why does it?

Memory accounting gets complicated very quickly once you peek a bit behind the curtains.

First of, a VM doesn't necessarily use all the memory right after boot. Linux VMs that don't need most of their memory for example. With the new line in the memory graph, you can see how it behaves over time (you need to enable the line by clicking on it in the legend).
Processes on the host that belong to the VM can need more memory in certain situations and then might free it up afterward. There is an overhead involved to handle the virtualized devices the VM sees.
The processes that belong to a VM are now accounted for with Proxmox VE 9. As of today, that it the KVM process itself, and the swtpm process if a TPM device is configured.

The Linux kernel reports back quite a lof of memory information for a process. We now use PSS, because it takes into account any memory that is shared between multiple processes, and calculates the usage for each process proportionally. The P in PSS. For example, if you have only one VM, then for that KVM process, all the necessary libraries need to be loaded. If you run multiple VMs, then the necessary libraries are already loaded, but all KVM processes make use of it.

I hope that sheds some light on it. It is quite a bit more complicated than used/cache/free on a per process level ;-)

https://docs.kernel.org/filesystems/proc.html