3 node cluster zfs & replication

[chris1284]

Hi,

i have a play around cluster of 3 nodes. i have tested ceph for storage and it was a fine user experience. easy to setup, nice performance with 1GbE and just some containers/small vms. than i found some posts on redit where they recommend zfs for small cluster with such bad ressources like 1gbE for ceph, smaller cpus, they call ceph a "beast" in functions (pro) and complexity (contra) if some one is interested https://www.reddit.com/r/Proxmox/comments/1qvyb4v/comment/o3pntfi/?context=3

Now decided to test the cluster with zfs and i deleted and uninstalled ceph, created my zfs pool on each node and and thought i can setup 1 or 2 replication tasks (Full pool from node 1 to 2 & 3).
Puff cake. I was relatively surprised that you have to set up one (actually 2 if you want to have the VM on each node) job for each VM/CT. If I scale this up to 20 vms/cts then I don't really want to do that.

my question where i could not find an answer.

1. is zfs with replication really the prefered soltuion from proxmox self for clusters (from the documentation a cluster should min have 3 nodes, 2 node cluster are deprecated and are only described in version 3x documentations)? i can not imagine that small businesses with three-digit count of vms/lxcs go this painfull way or that proxmox would recommedn this

2. if we have such a cluster, is there a way to fully replicate the entire pool to all nodes?

thx and greetings

 

[ness1602]

1. Yes it is because you don't have shared storage for all your nodes, which is usually ceph.
2. There are some aftermarket scripts to replicate whole pool, but you usually want to replicate different machines with different schedules, eg dbs in 1-5m, app servers in /15 min,etc etc.

If you don't have >=2.5gb interconnection between nodes i wouldn't even be thinking about ceph,so replicate nodes1 and 2 to node3, set up HA and it is what it is.

 

[pvps1]

first of all: 10gbit as that cheap that i would not run anything with lower bandwidth

2. you should design your setup/cluster depending on your internal and external SLA

knowing what you must achive you can find the correct technical solutions.

e.g. we do not activate HA on any clusters, have several storage tiers (local zfs, shared storage ssd, shared storage spinning....)

your example: if you need replication of complete pools I would consider shared storage. but do you really need it? or would AD/DNS replicated be enough (as example)

 

[bl1mp]

Hi Chris 1284,

maybe you can consider, a 3-Node meshed Ceph cluster. The 10G NICs are cheap and you can save the money for expensive switching infrastructure.

Yes, Ceph has many many useful features.
But it is super easy to setup a working 3-node ceph cluster from the webui. So the complexity for that use case is quite low.

BR, Lucas