Node Using More RAM Than Assigned

[Misinthe]

Hello everyone,

I am having an issue with my Proxmox Node, I have a total of 192GB (188.84GB usable), within all my VM’s I have assigned 94GB but the system Summary says it is using 173.49GB (91.86%), I don’t understand how this is happening, does anyone have any idea what do I need to check or how I can fix this issue? Thank you.

 

[LnxBil]

Have you set disk caching (per disk in every VM) to something else than "none"?

Each VM uses more RAM than assigned .. it's normal and depends on the maximum memory how much more for the controlling processes etc.

There is also a lot of disk caching on the PVE side involved, so using all the RAM for caching is the desired state in every OS. On Linux, used RAM for cache is shown as "used", whereas it is shown as "free" in Windows.

 

[Dunuin]

And in case you are using ZFS it also might use up to 50% of your total hosts RAM for caching (see "ARC").

 

[Max Carrara]

Hi Misinthe!

As Dunuin already pointed out, the ARC (or Adaptive Replacement Cache) of ZFS might use up to half of your totally available RAM on your machine as per the default settings of ZFS. To view how much memory the ARC is currently using, you can run the arcstat command as root on your host system.

Also, it's worth to note that this isn't actually an issue per se and is expected of ZFS. The ARC is a very efficient cache for the ZFS filesystem, so any virtual disk that sits on top of ZFS is automatically cached by the ARC by default.

However... if you want to turn off the ARC, which I do not recommend, you can certainly do so:

To check if the ARC is enabled for your dataset:
zfs get primarycache <your_pool>/<dataset>

You should see the VALUE to be set to all.


To disable the ARC for your dataset:
zfs set primarycache=none <your_pool>/<dataset>

NOTE: Turning the ARC off on your actual host might cause a huge decrease in performance. I highly suggest you don't try this on your host. You have been warned. After doing this, it may be necessary to reboot your host in order for the setting to apply.


To re-enable the ARC for your dataset:
zfs set primarycache=all <your_pool>/<dataset>

You might need to reboot after this as well.


If you really want to try this and play around safely, you can try to setup PVE within your PVE host using nested virtualization.

Hope that could clear everything up!

 

[Dunuin]

Jup, disabling the ARC is really not recommended. If you think 50% is too much, its also possible to limit the ARC size: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage

 

[Misinthe]

Have you set disk caching (per disk in every VM) to something else than "none"?

Each VM uses more RAM than assigned .. it's normal and depends on the maximum memory how much more for the controlling processes etc.

There is also a lot of disk caching on the PVE side involved, so using all the RAM for caching is the desired state in every OS. On Linux, used RAM for cache is shown as "used", whereas it is shown as "free" in Windows.

I have not, how do I check for this?

And in case you are using ZFS it also might use up to 50% of your total hosts RAM for caching (see "ARC").

I do have my 4TB raid running ZFS.

-snip-

Thank you for all this information, I guess I only have 1 more follow up question, if I need to take some of that RAM to use in new VMs, will that create a negative impact in the system, or the ZFS will just have less cache?

 

[Max Carrara]

Thank you for all this information [...]

You're welcome!

[...] if I need to take some of that RAM to use in new VMs, will that create a negative impact in the system, or the ZFS will just have less cache?

The ZFS ARC will shrink and grow dynamically depending on your system's needs. If some other process in your system requires RAM that's taken by the ARC, the ARC will be asked to shrink. So yes, ZFS will just have less cache.

If your VMs use the majority of your system's RAM, it might have a negative impact on performance, but when that happens depends on your system's hardware (from my experience). Usually it's not a thing to worry about as long as you leave some RAM to ZFS.

Overall it's good to just let ZFS do its thing - ZFS grabbing your "spare RAM" is very positive in all aspects, as the ARC usually outperforms your system's LRU cache.

 

[Misinthe]

You're welcome!



The ZFS ARC will shrink and grow dynamically depending on your system's needs. If some other process in your system requires RAM that's taken by the ARC, the ARC will be asked to shrink. So yes, ZFS will just have less cache.

If your VMs use the majority of your system's RAM, it might have a negative impact on performance, but when that happens depends on your system's hardware (from my experience). Usually it's not a thing to worry about as long as you leave some RAM to ZFS.

Overall it's good to just let ZFS do its thing - ZFS grabbing your "spare RAM" is very positive in all aspects, as the ARC usually outperforms your system's LRU cache.

Awesome, thank you very much to all, I understand now, I was worried about spinning more VM’s but I should be good.

 

[Max Carrara]

Awesome, thank you very much to all, I understand now, I was worried about spinning more VM’s but I should be good.

Glad we could help you! Please mark your thread as solved