ZFS or not ZFS ?

B

b416

Member
Dec 23, 2021
14
2
8
60
I have a small OVH server with PVE 7.1 installed and a douzain of VMs on it. It has 64Gb of RAM and 2 mirrored 450Gb NVMe disks.

It was working Ok, but pretty packed up, not much space left for new VMs.

I saw an opportunity with Christmas sales and took 2 new servers with a better CPU and 128Gb of RAM, same disks. I migrated manually (backup/copy/restore) all my VMs to one of the servers (they are in a cluster) and I noticed something odd : where on the old server my VMs used about 50Gb of RAM, I only stared 5 of them and they are using 95Gb of RAM !

The only true difference between the hosts is that I formatted the data partition on the new one with ZFS, instead of LV volume on the old. I read that ZFS takes some more RAM to work, but I didn't think it was that a much.

My servers will be working in a cluster, but a poor man's cluster, I do not have shared storage and will not be able to use HA. Besides that there is also the IP address problem, I have a block of 16 IP adresses on each server, but I have to switch them manually in the OVH admin panel if I want to move a VM from one server to the other.

So my question is : is there any real advantage for me to use ZFS in this case, or is it just a waste of ressources ?
 
b416 said:
The only true difference between the hosts is that I formatted the data partition on the new one with ZFS, instead of LV volume on the old. I read that ZFS takes some more RAM to work, but I didn't think it was that a much.

My servers will be working in a cluster, but a poor man's cluster, I do not have shared storage and will not be able to use HA. Besides that there is also the IP address problem, I have a block of 16 IP adresses on each server, but I have to switch them manually in the OVH admin panel if I want to move a VM from one server to the other.

So my question is : is there any real advantage for me to use ZFS in this case, or is it just a waste of ressources ?
Click to expand...
ZFS by default will use up to 50% of your RAM for its ARC for caching. But that doesn't mean that you will loose 50% of your RAM. If something needs the RAM, your ARC will shrink to get some free RAM. The ARC will always need some GB for a fast operation but not all the 64GB. You can run arc_summary to get information on your ARC usage and performance.

There are alot of ZFS features you can make use of. Deduplication, block level compression, replication between nodes, bit rot protection so your VMs aren't silently corrupting over time any longer, ...
 
Thanks for the response.

I have another question : does it make sense to put the ZFS on a soft RAID1 partition or can I use RAID0 (and double the space) safely ?
 
It does not make sense to put ZFS on any type of RAID. It is better if you let ZFS do mirror or stripe itself. Note that ZFS can detect errors but not repair them if the data has no copies like RAID0.
 
Ok, so the best option is to create ZFS with the mirror option and assign my 2 partitions from my 2 NVMe disks ?
 
ZFS shouldn't be run ontop of any SW/HW raid. ZFS needs direct access to the physical drives without any abstraction layer in between. So yes, choose a ZFS mirror and use the whole drives or partitions.
 
Dunuin said:
ZFS by default will use up to 50% of your RAM for its ARC for caching.
Click to expand...
For those with 128GB RAM systems ending up here through a Google search and frowning upon this: the above used to be true, but no longer stands correct.
From https://pve.proxmox.com/wiki/ZFS_on_Linux:
ZFS uses 50 % of the host memory for the Adaptive Replacement Cache (ARC) by default. For new installations starting with Proxmox VE 8.1, the ARC usage limit will be set to 10 % of the installed physical memory, clamped to a maximum of 16 GiB.
Click to expand...
 
  • Like
Reactions: Dunuin
You must log in or register to reply here.
Top Bottom