Smartest Way (Sanoid/Syncoid?) to Replicate Data Across ZFS Cluster (Non-Ceph/GlusterFS)

[pr0927]

Hi all, so I setup three Proxmox servers (two identical, one "analogous" - and the basics about the setup is as follows:

• VMs and CTs are replicated every 15 minutes from one node to the other two.
• One CT runs Cockpit for SMB shares - bind-mount to the drive pool with the datasets that are SMB-shared.
  • I use this for accessing folders via GUI over the network from my PC.
• One CT runs an NFS server (no special GUI, only CLI) - bind-mount to the drive pool with the datasets that are NFS-shared (same as SMB-shared ones).
  • Apps that need to tap into data, use NFS shares (such as Jellyfin, Immich, Frigate) provided by this CT.
• Two VMs are of Debian, running Docker for my apps.
• VMs and CTs are all stored on 2x2TB M.2 NVMe SSDs.
• Data is stored in folders (per the NFS/SMB shares) on a 4x8TB ZFS pool with specific datasets like Media, Documents, etc. and a 1x4TB SSD ZFS "pool" for Frigate camera footage storage.

Due to having hardware passed-through to the VMs (GPU and Google Coral TPU) and using hardware resource mappings (one node as an Nvidia RTX A2000, two have Nvidia RTX 3050s - can have them all with the same mapped resource node ID to pass-through without issue despite being different GPUs), I don't have instant HA failover.

Additionally, as I am using ZFS with data on all three separate nodes, I understand that I have a "gap" window in the event of HA where the data on one of the other nodes may not be all the way up-to-date if a failover occurs before a replication.

So after all the above - this brings me to my question - what is the best way to replicate data that is not a VM or a CT, but raw data stored on those ZFS pools for the SMB/NFS shares - from one node to another?

I have been using Sanoid/Syncoid installed on one node itself, with cronjobs. I'm sure I'm not using it perfectly (boy did I have a "fun" time with the config files), and I did have a headache with retention and getting rid of ZFS snapshots in a timely manner to not fill up the drives needlessly - but it seems to be working.

I just setup the third node (the "analogous" one in specs) which I want to be the active "primary" node and need to copy data over from the other current primary node - I just want to do it intelligently, and then have this node, in its new primary node role, take over the replication of data to the other two nodes.

 

[pr0927]

Sorry to bump, not sure if no response on this because I presented my question in some stupid or unwelcome fashion, or if my question has somehow perplexed many (unlikely!) - but still hoping for some insight from those smarter than me.

 

[UdoB]

no response on this

There are just too many possible answers ;-)

I am old, so I would either opt for classic and simple "rsync" or ZFS snapshot replication. There are several tools available for that - and to obtain some experience with it is completely up to you

Have fun!

 

[pr0927]

There are just too many possible answers ;-)

I am old, so I would either opt for classic and simple "rsync" or ZFS snapshot replication. There are several tools available for that - and to obtain some experience with it is completely up to you

Have fun!

Hmm, what would you say the pros and cons are of either method?

If I stick to ZFS replication, is this the smart way to do it?

 

[UdoB]

If I stick to ZFS replication, is this the smart way to do it?

Sure. As long as this is a separate dataset it is safe+secure and it works fast - it can be run frequently. The integrity of the whole copy process is guaranteed by ZFS - snapshots are an atomar function. Rsync can not really do that, as far as I know.

The only downside I see is that you can not cherry-pick which specific files are sync'ed.

 

[pr0927]

Sure. As long as this is a separate dataset it is safe+secure and it works fast - it can be run frequently. The integrity of the whole copy process is guaranteed by ZFS - snapshots are an atomar function. Rsync can not really do that, as far as I know.

The only downside I see is that you can not cherry-pick which specific files are sync'ed.

Oh, O.K. - do I want to do it with one node running it and sending to the other two, or should I be somehow configuring each node to create snapshots and send to each other, so that if one takes over it continually, without my intervention, also does the syncing of data to the other nodes?

Is this even possible, or would the snapshots conflict with one another?

 

[UdoB]

Is this even possible, or would the snapshots conflict with one another?

I do not use such a construct, so I could only guess. So my recommendation is: just go an try it - you learn things by doing and probably it will "just work".

Please report back your findings

 

[pr0927]

I do not use such a construct, so I could only guess. So my recommendation is: just go an try it - you learn things by doing and probably it will "just work".

Please report back your findings

Hahaha, well I would, but I don't want to risk data loss or anything funny - I rely on this data for my self-hosted apps, which are important to me (as is the data. I don't want to suddenly overwrite stuff or lose items (or cause corruption).

Surely you're not the only one on this entire forum with suggestions? Has nobody done ZFS replications like this, or considered multi-node replications to each other?