understanding ZFS RAM usage

[Binary Bandit]

Hi All,

Is there a command or two that illustrates where RAM is being consumed on our Proxmox systems that are using ZFS?

For example, here is the RAM usage on a new system:



Column B is my math showing expectations with my current understanding of what is using RAM ... This server has 128MB or RAM; we've allocated 32GB to one VM "DC," 32GB to another "Exchange," and 32GB to the ARC cache. Without setting aside a couple of GB for Proxmox, we should have 29.15GB free at a minimum.

Column C is the current reality ... ARC using 30GB, Exchange 13.78, and DC 7.94. If I remove that from the 98.04 that Proxmox shows as used (in the server's GUI summary tab), I'm left wondering where the 46.32GB is used/how I can account for it.

So that's the first question. Where is the 46.32 GB being used, and is there a command(s) to show it? Arcstat isn't telling the whole story.

Second question, how should I manage the 17.17 GB of overallocation in column D? Is there enough room/time for ARC to back off and free up RAM without a VM being terminated? "options zfs zfs_arc_min" is set to 8GB, so I believe that I've left room for this to happen. I have seen instances where a VM PID is terminated before ARC releases RAM, and I am wondering what the best way to manage this is ... or do I need to restrict ARC / VM max RAM further?

best,

James

 

[leesteken]

Please account for the filesystem cache inside the VMs. Cache inside the VMs is usually not counted as used by the operating system inside the VM because it can be freed/reused at a moment's notice. Proxmox however does not know what the memory is used for and will count it as used.

... we've allocated 32GB to one VM "DC," 32GB to another "Exchange," and 32GB to the ARC cache. ...

... the 98.04 that Proxmox shows as used (in the server's GUI summary tab), ...

You have 2x32GB VMs plus 32GB ARC, which is 96GB and there is some overhead and some memory used by the Proxmox software. Proxmox reports 98GB used, which matches that 96GB+overhead quite nicely, if I'm not mistaken.

 

[Binary Bandit]

Thanks leesteken, that checks out. Here are updated numbers for column B:


For those who find this thread, I obtained the updated Exchange and DC VM numbers by opening Windows task manager on each and adding the RAM "In use" to "Cached". Now I can see that the numbers line up. The remaining difference between 29.15 and 25.9 GB free is just Proxmox itself and ZFS.

Does anyone have thoughts on how to manage the overallocation? ... see the last paragraph of the OP for clarification.

 

[leesteken]

For those who find this thread,

... it's always filesystem cache . See also the other threads about memory usage and discrepancies. And there is always a little memory overhead for emulating devices and keeping track of things (that use memory).