[SOLVED] ZFS-RAID10 vs Software RAID10 in a RAM limited environment

[Razva]

Hello,

I'm deploying a new cluster with 3 machines, each one having 64GB RAM + 2x512GB NVMe SSD. The VMs that will run on the cluster will use about 70-80% of available RAM, so there will be little spare RAM left (4-8 GB).

I would like to use ZFS, as it's supported by Proxmox, but I'm afraid about how ZFS will perform with limited RAM. In my tests ZFS used even 50% of existing memory.

Here are some questions:

1) Can lack of RAM result in data loss in a ZFS pool?

2) Should I limit RAM usage, or should I just allow ZFS to use as much RAM as possible?

3) In case the node's memory is 100% used (because of ZFS), will this affect running VMs? Or ZFS will clear/unlock necessary resources in order for the VMs to properly start/run?

4) In case the node is "out of memory" (because of ZFS "over-usage"), will I be able to create new VMs?

3) For this specific scenario, do you advise me to go with "limited RAM ZFS-RAID10" or do a Debian Software RAID1 and install Proxmox & LVM Thin on top of that?

Thank you!

 

[Walery]

https://pve.proxmox.com/wiki/ZFS_on_Linux

Limit ZFS Memory Usage​

It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent performance shortage of the host. Use your preferred editor to change the configuration in /etc/modprobe.d/zfs.conf and insert:
options zfs zfs_arc_max=8589934592
This example setting limits the usage to 8GB.

If your root file system is ZFS you must update your initramfs every time this value changes: # update-initramfs -u

 

[Razva]

https://pve.proxmox.com/wiki/ZFS_on_Linux

Limit ZFS Memory Usage​

It is good to use at most 50 percent (which is the default) of the system memory for ZFS ARC to prevent performance shortage of the host. Use your preferred editor to change the configuration in /etc/modprobe.d/zfs.conf and insert:
options zfs zfs_arc_max=8589934592
This example setting limits the usage to 8GB.

If your root file system is ZFS you must update your initramfs every time this value changes: # update-initramfs -u

Thank you for your answer, I've already read the documentation provided by Proxmox, along with some other tutorials/videos.

Unfortunately the documentation doesn't provides answers to my initial questions.

 

[Walery]

Thank you for your answer, I've already read the documentation provided by Proxmox, along with some other tutorials/videos.

Unfortunately the documentation doesn't provides answers to my initial questions.

If you leave the default settings. ZFS will use 50% of the memory - this will be 32GB
If there is not enough memory then it will use SWAP, but it is slow.
You need to limit your ZFS memory usage. It is obvious.

 

[Razva]

If you leave the default settings. ZFS will use 50% of the memory - this will be 32GB
If there is not enough memory then it will use SWAP, but it is slow.
You need to limit your ZFS memory usage. It is obvious.

Thank you for your reply, but kindly read the OP.