ZFS multi node cluster?

[Magneto]

Is it possible to setup a ZFS multi node cluster, like one would do with CEPH?

Let's say I have 20x servers, each with 24 SSD's, and I want to use ZFS on each server. Can I have replicated storage across the cluster, so that in the event that a node fails, or I need to move some VM's off one node to another, it will happen online?

And is it possible to scale such a cluster?

I have searched the forum but cannot find anything on this, so I am just wondering if anyone has done it before?

 

[H4R0]

It's possible but definitely not something you want.

Ceph uses multicast so it sends once and all nodes receive.

ZFS only sends to a single node and repeats the step for each so you get 20x the traffic.

ZFS also is not live, it sends in intervals using snapshot replication with minimal interval of 1 minute.

It's like a power outage in that regard for the 1 minute frame. So never use ZFS storage replication, it has very limited use cases.


You can use ZFS over iSCSI with a SAN/NAS or use Ceph/GlusterFS for hyper converged storage.

GlusterFS can be used on ZFS but you will end up with qcow on GlusterFS on ZFS, so double cow which is bad.

Ceph is the way to go in your case.

 

[Magneto]

Thank you for the explanation. CEPH is the better option then.