Proxmox 6.x consumes more memory than assigned using ZFS

[Joris L.]

After being faced a few times with extraordinary memory consumption i took a look at the shell using glances.


out of principle i never count on swap space and don't configure it since my use cases don't call for it.

While the RAM assigned is correctly estimated proxmox/kvm decided to consume a lot more. How to disable this ?

 

[dcsapak]

While the RAM assigned is correctly estimated proxmox/kvm decided to consume a lot more. How to disable this ?

what exactly does use memory? where do you check how much memory is used?

do you use zfs? by default it uses up to half of your memory

 

[Joris L.]

what exactly does use memory? where do you check how much memory is used?

do you use zfs? by default it uses up to half of your memory

yes, running ZFS, to my memory with low memory consumption configured.

from a shell I run glances and notice the assign memory is for example 5GB but the VIRT memory is 12GB which is cumbersome as it is what proxmox appears to really consume in terms of resources instead of the 5GB assigned.

Same for any other tool i checked such as top, htop etc.

Not certain why this is so really. I allocate about 60GB but proxmox report 76 out of 78GB is in use.

 

[dcsapak]

i'd check top/htop to see which processes consume the memory

 

[Joris L.]

i'd check top/htop to see which processes consume the memory

? ALL of them ?

Somehow proxmox consumes consistenly more memory than allocated. It is normal after a reboot, after a few vm have restarted it builds up and only goes down again after a restart of shutting down all vm and starting them again etc.

 

[H4R0]

Checkout my thread.

https://forum.proxmox.com/threads/windows-vm-using-more-memory-then-allocated.79368/#post-351549

 

[Joris L.]

Checkout my thread.

https://forum.proxmox.com/threads/windows-vm-using-more-memory-then-allocated.79368/#post-351549

Thanks

Sadly, since ballooning causes more problems than it solves, i only use ballooning for just one VM.

Even when disabled (now) memory is at 94% while effective reservation is around 65%

 

[Joris L.]

Yesterday i restarted ALL VM without restarting the server and after disabling all ballooning for memory. I've not observed excessive memory usage since.

Prior to the VM restarts the overconsumption of memory whas noticeable throughout all VM, not just the single one where ballooning was enabled. This may hint at an issue with memory management which may expose VM and/or host to memory related weaknesses.

 

[Joris L.]

My impression is the ballooning may somehow negatively interplay with KSM sharing.

 

[H4R0]

My impression is the ballooning may somehow negatively interplay with KSM sharing.

KSM starts at 80% memory usage, I'm below that and still have the problem.

 

[Joris L.]

KSM starts at 80% memory usage, I'm below that and still have the problem.

Interesting. Are you absolutely sure about KSM ? My system showed over 10GB KSM out of 80GB total with just over 65% memory usage.

 

[H4R0]

Interesting. Are you absolutely sure about KSM ? My system showed over 10GB KSM out of 80GB total with just over 65% memory usage.

Yes the default of /etc/ksmtuned.conf is KSM_THRES_COEF=20, which equals to at least 80% memory usage (less then 20% free).

I always have 0% KSM usage unless I'm above the threshold.

Maybe you tuned your values ?

 

[Joris L.]

Yes the default of /etc/ksmtuned.conf is KSM_THRES_COEF=20, which equals to at least 80% memory usage (less then 20% free).

I always have 0% KSM usage unless I'm above the threshold.

Maybe you tuned your values ?

That file /etc/ksmtuned.conf is pristeen, all entries are commented. Which probably has the same outcome as the configuration you mention ?

 

[H4R0]

That file /etc/ksmtuned.conf is pristeen, all entries are commented. Which probably has the same outcome as the configuration you mention ?

Yes its the default values then, can you post a screenshot of your node summary ?

 

[Joris L.]

obvioiusly, as we spoke it had to go near 80%

 

[H4R0]

obvioiusly, as we spoke it had to go near 80%

View attachment 21527

Yeah your server does KSM because it hit the 80% threshold then deduplicated 10GB of memory and thus the memory usage went down.

Without KSM you would be at 93% memory usage.

 

[Joris L.]

Yeah your server does KSM because it hit the 80% threshold then deduplicated 10GB of memory and thus the memory usage went down.

Without KSM you would be at 93% memory usage.

Not my impression. It showed 10GB KSM with 67% reported use as well. For reasons beyond me, while updating the forum the reported memory rose to 79%. The total of VM assigned account for much less than 93% of RAM, roughly 47GB of RAM is configured for all VM combined. Note i have stopped a 4GB VM in the meantime.

However i look or calculate based on what i see in the shell, the total sum of memory used does not rise above 70% (VM + system + zfs etc.)

 

[Joris L.]

the "issue" is back, no machine is running with ballooning enabled. edit: Note the server was not restarted, just all the VM, as such ZFS memory consumption is possibly not part of the growing memory consumption.

Afaik it is not because of the memory reservation reported in MEM but that reported in VIRT. As such i noticed the 3 elastic beats i'm running on the server to consume 12GB combined in VIRT while just 1GB combined in MEM.

 

[allocacoc]

I'm having same behaviour here. I have a bare metal with 2 x1TB VNMe set as ZFS (RAID 1) and 64GB RAM.
I'm running 6 VMs with ballooning enabled for a supposed total of 26GB of RAM.

RAM graph shows 49 GB of usage and no KSM sharing.



Is there a way to know what is consuming the other 23GB of RAM? Htop shows that only VMs are using RAM, and it doesn't totalize 49GB.
And now I wonder, memory will be made available if I run more VMs? Or there are actually only 13 GB left for next VMs?

I apologize if I ask the obvious, but it's my first time with ProxMox and ZFS.

 

[Joris L.]

I'm having same behaviour here. I have a bare metal with 2 x1TB VNMe set as ZFS (RAID 1) and 64GB RAM.
I'm running 6 VMs with ballooning enabled for a supposed total of 26GB of RAM.

RAM graph shows 49 GB of usage and no KSM sharing.

View attachment 21681

Is there a way to know what is consuming the other 23GB of RAM? Htop shows that only VMs are using RAM, and it doesn't totalize 49GB.
And now I wonder, memory will be made available if I run more VMs? Or there are actually only 13 GB left for next VMs?

I apologize if I ask the obvious, but it's my first time with ProxMox and ZFS.

Hi,

Yes, this looks very similar to what i'm observing.

Reasoning is ZFS reserves half of available RAM, looking at the memory consumption i have not found such to be so. Also, in your cases that would put the RAM consumed at between 57-58GB usage.

What i'm considering is KSM may actually be the issue, not ballooning.

If you feel like it consider disabling KSM for some time, at worst it coud but should not boost RAM consumption beyond 58GB.

https://pve.proxmox.com/wiki/Dynamic_Memory_Management


Disable KSM
If you don't care about memory optimization but care about save CPU overhead produced by KSM, in Proxmox >= 4.x you can disable it with:

systemctl disable ksmtuned
and reboot