RAM usage well exceeds VM assignment

[Sandbo]

Hi,

Lately I realized that the RAM usage on PVE exceed the amount I allocated for the VMs.
For example, now I am running a single VM and 4 GB RAM has been allocated to it, but the summary of the host says 15GB+ RAM has been used.

I am running PVE 6 with 32 GB RAM and a ZFS storage pool (1.8 TB mirror)

Host summary:

VM summary:

htop somehow shows multiple instance of the same VM and each takes 13.8% of RAM. Is that expected?
13.8% of 32 GB RAM is ~ 4GB.

 

[BobhWasatch]

There will be a KVM thread for each core you gave the VM, plus some extra ones to support virtual hardware. All of mine have two additional, same as yours.

IIRC, ZFS uses up to 1/2 of your RAM for ARC cache. See here:
https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage

I have only limited experience with ZFS and am not currently using it. Hopefully someone more knowledgeable will chime in about ZFS memory usage.

 

[Sandbo]

There will be a KVM thread for each core you gave the VM, plus some extra ones to support virtual hardware. All of mine have two additional, same as yours.

IIRC, ZFS uses up to 1/2 of your RAM for ARC cache. See here:
https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage

I have only limited experience with ZFS and am not currently using it. Hopefully someone more knowledgeable will chime in about ZFS memory usage.

Thanks for the ideas, that explained a lot regarding the multiple threads.
I believe the ZFS might be eating up quite a bit of RAM, but then I was expecting to see them in htop which didn't seem to be the case here.

 

[LnxBil]

I believe the ZFS might be eating up quite a bit of RAM, but then I was expecting to see them in htop which didn't seem to be the case here.

htop basically only shows user land stuff, ZFS and other filesystems are in the kernel space so their memory requirements are not shown here. Also, all space used by filesystem caching (old school block cache and ZFS ARC) is seen as "used" in Linux (and many other OSes) and not as "free" as in Windows. Therefore you have the seen discrepancy of the sum of all VMs memory. Please also refer to https://www.linuxatemyram.com/

 

[Sandbo]

htop basically only shows user land stuff, ZFS and other filesystems are in the kernel space so their memory requirements are not shown here. Also, all space used by filesystem caching (old school block cache and ZFS ARC) is seen as "used" in Linux (and many other OSes) and not as "free" as in Windows. Therefore you have the seen discrepancy of the sum of all VMs memory. Please also refer to https://www.linuxatemyram.com/

Thanks for the explanation.
It would be great if we can find out the RAM used or even control how much cache is allocated, as with the current behaviour I could run into issue with assigning RAM to the VMs while I thought I had sufficient.

Is there a way to design the size of the cache?

 

[LnxBil]

Is there a way to design the size of the cache?

@BobhWasatch already answered that:
https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage

 

[Sandbo]

@BobhWasatch already answered that:
https://pve.proxmox.com/wiki/ZFS_on_Linux#_limit_zfs_memory_usage

Thanks, missed that.