RAM usage does not add up

[garasmell]

Host has 104 GB (DDR3 ECC) and about 88 GB is being used (according to the node's summary page). The running VMs themselves are configured for 56 GB total and only 40 GB of that is actually used at the moment (according to each VM's summary page). Why is there a 48 GB difference in host current usage vs total VMs' current usage?

P.S. - Only these 3 VMs are running. There are no LXC containers/instances running/set up. 2 of the VMs have GPU passthrough, if that matters (configured for 16 GB of RAM each). The VMs are stored on ZFS (4 x M.2 NVMe SSDs in RAIDZ1) and the Proxmox installation on a separate SATA SSD.

 

[cwt]

Often asked and answered: ZFS uses RAM for caching. If this is not a new (V8) installation, ZFS can consume up to 50% of all available RAM (arc). You can limit the arc size if you want. To display the arc usage run

arc_summary

from your console.

 

[leesteken]

Depending on whether QEMU Guest Agent and/or the VirtIO Balloon device/driver is active the reported VM usage may or may not match the reported amount from the operating system inside the VM. Try to not over-commit on memory and assume that each VM actually takes all of the memory you set it to. And then Proxmox and ZFS also need memory.

2 of the VMs have GPU passthrough, if that matters (configured for 16 GB of RAM each).

Because of DMA, every VM with PCI(e) passthrough needs to all of the VM memory pinned into actual host memory (and ballooning therefore does not work).

The VMs are stored on ZFS

Make sure to limit ZFS ARC: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage

 

[garasmell]

Depending on whether QEMU Guest Agent and/or the VirtIO Balloon device/driver is active the reported VM usage may or may not match the reported amount from the operating system inside the VM. Try to not over-commit on memory and assume that each VM actually takes all of the memory you set it to. And then Proxmox and ZFS also need memory.

Because of DMA, every VM with PCI(e) passthrough needs to all of the VM memory pinned into actual host memory (and ballooning therefore does not work).

Make sure to limit ZFS ARC: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_zfs_limit_memory_usage

Often asked and answered: ZFS uses RAM for caching. If this is not a new (V8) installation, ZFS can consume up to 50% of all available RAM (arc). You can limit the arc size if you want. To display the arc usage run

arc_summary

from your console.

WOOOOOOOOW!! That's a huuuuuge caveat! Makes me re-think running ZFs on Proxmox, even with limiting the arc size. Thanks everyone!

 

[UdoB]

Makes me re-think running ZFs on Proxmox, even with limiting the arc size.

Really? Each and every filesystem needs buffers and caches. ZFS is just more clever and makes potentially use of more RAM. (You may shrink it drastically...)

If you are going to drop ZFS read this first: https://forum.proxmox.com/threads/question-about-file-system.159762/post-733718

 

[garasmell]

Really? Each and every filesystem needs buffers and caches. ZFS is just more clever and makes potentially use of more RAM. (You may shrink it drastically...)

If you are going to drop ZFS read this first: https://forum.proxmox.com/threads/question-about-file-system.159762/post-733718

Nah, I'm not going to drop ZFS; just frustrated at discovering another ZFS tax. Been hoping to downsize my infrastructure, but ish like this keeps it bloated