ZFS or not ZFS ?

[b416]

I have a small OVH server with PVE 7.1 installed and a douzain of VMs on it. It has 64Gb of RAM and 2 mirrored 450Gb NVMe disks.

It was working Ok, but pretty packed up, not much space left for new VMs.

I saw an opportunity with Christmas sales and took 2 new servers with a better CPU and 128Gb of RAM, same disks. I migrated manually (backup/copy/restore) all my VMs to one of the servers (they are in a cluster) and I noticed something odd : where on the old server my VMs used about 50Gb of RAM, I only stared 5 of them and they are using 95Gb of RAM !

The only true difference between the hosts is that I formatted the data partition on the new one with ZFS, instead of LV volume on the old. I read that ZFS takes some more RAM to work, but I didn't think it was that a much.

My servers will be working in a cluster, but a poor man's cluster, I do not have shared storage and will not be able to use HA. Besides that there is also the IP address problem, I have a block of 16 IP adresses on each server, but I have to switch them manually in the OVH admin panel if I want to move a VM from one server to the other.

So my question is : is there any real advantage for me to use ZFS in this case, or is it just a waste of ressources ?

 

[Dunuin]

The only true difference between the hosts is that I formatted the data partition on the new one with ZFS, instead of LV volume on the old. I read that ZFS takes some more RAM to work, but I didn't think it was that a much.

My servers will be working in a cluster, but a poor man's cluster, I do not have shared storage and will not be able to use HA. Besides that there is also the IP address problem, I have a block of 16 IP adresses on each server, but I have to switch them manually in the OVH admin panel if I want to move a VM from one server to the other.

So my question is : is there any real advantage for me to use ZFS in this case, or is it just a waste of ressources ?

ZFS by default will use up to 50% of your RAM for its ARC for caching. But that doesn't mean that you will loose 50% of your RAM. If something needs the RAM, your ARC will shrink to get some free RAM. The ARC will always need some GB for a fast operation but not all the 64GB. You can run arc_summary to get information on your ARC usage and performance.

There are alot of ZFS features you can make use of. Deduplication, block level compression, replication between nodes, bit rot protection so your VMs aren't silently corrupting over time any longer, ...

 

[b416]

Thanks for the response.

I have another question : does it make sense to put the ZFS on a soft RAID1 partition or can I use RAID0 (and double the space) safely ?

 

[leesteken]

It does not make sense to put ZFS on any type of RAID. It is better if you let ZFS do mirror or stripe itself. Note that ZFS can detect errors but not repair them if the data has no copies like RAID0.

 

[b416]

Ok, so the best option is to create ZFS with the mirror option and assign my 2 partitions from my 2 NVMe disks ?

 

[Dunuin]

ZFS shouldn't be run ontop of any SW/HW raid. ZFS needs direct access to the physical drives without any abstraction layer in between. So yes, choose a ZFS mirror and use the whole drives or partitions.

 

[LnxBil]

So my question is : is there any real advantage for me to use ZFS in this case, or is it just a waste of ressources ?

You poor man's cluster idea relyies on this. You need ZFS replication for this, don't you?