How much ram for ZFS and PVE?

[Alessandro 123]

Our new host has 128GB RAM.
Currently, all of our VMs are stored on XenServer hosts with max 64GB RAM.
This new server has twice the amount.

How much RAM should I set for ZFS ? 16GB ? And what about PVE ? Is 16GB enough ?

Doing this allow me to keep 96GB free for VMs

Silly question: if I add a Gluster storage, is PVE able to cache the VM disk in RAM lowering the network overhead caused by the network filesystem ? If yes, probably, keeping 32GB available for PVE would be better.

 

[wolfgang]

Hi,

PVE about 1GB
ZFS base about 4GB and 1GB for each TB used disc space.
this is without dedup or L2ARC

 

[Alessandro 123]

I have less than 1TB of storage.

But is PVE able to 'cache' locally (in RAM) the Gluster storage?

 

[Rhinox]

What I did is I gave ZFS as much RAM as I could while my PVE is guaranteed NOT to use swap at all (even if all VMs are running and using all vRAM they have). In my case it is 10GB (out of 64GB), while my storage-size is ~1TB (SSD-only)...

 

[Alessandro 123]

VMs are not an issue, out current max use of RAM by the VMs are less than 64GB, thus the other 64GB are for ZFS and PVE

 

[Rhinox]

Then give at least ~30GB to zfs. When you have that much RAM, use it. And any using of RAM is better than not using it at all. You could also give more to VMs, i.e. for /tmp in tmpfs (ram-disk)...

 

[Alessandro 123]

No all VMs are stored on local ZFS. Most of them are stored in gluster, that's why I'm asking if PVE (or, better, linux) is able to locally cache mostly used blocks from a network FS.

 

[Nemesiz]

Look at gluster documentation. Try this
https://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning
https://access.redhat.com/documenta...n_Guide/chap-User_Guide-Managing_Volumes.html

 

[Alessandro 123]

Posted docs talks about optimizing the gluster node.
i'm talking about optimizing the PVE node. My gluster servers are located externally to the PVE.

PVE is only used as hypervisors. I'm not running gluster on the same PVE server

 

[Nemesiz]

https://gluster.readthedocs.io/en/latest/Quick-Start-Guide/Architecture/#architecture

look down to Types of Translators and you will see The default / general hierarchy of translators in vol files

I don`t have where to test but client side is tunable

 

[Alessandro 123]

PVE doesn't use fuse but direct access to gluster via libgfapi integrated in qemu

So, what can I tune on client side to limit the network usage?

I don't know if Linux is able to locally cache a remote filesystem. In case of NFS, it should be able to store the most accessed files locally in RAM, but what about gluster?
We don't have any file access, only blocks

Probably, I have to cache most accessed files inside each VM by adding RAM as much as possible. RAM is from the PVE node. The VM load a file and store in on RAM, this is already an optimization as further requests to that file doesn't need to go through the network.