Smartest Way (Sanoid/Syncoid?) to Replicate Data Across ZFS Cluster (Non-Ceph/GlusterFS)

[pr0927]

Hi all, so I setup three Proxmox servers (two identical, one "analogous" - and the basics about the setup is as follows:

• VMs and CTs are replicated every 15 minutes from one node to the other two.
• One CT runs Cockpit for SMB shares - bind-mount to the drive pool with the datasets that are SMB-shared.
  • I use this for accessing folders via GUI over the network from my PC.
• One CT runs an NFS server (no special GUI, only CLI) - bind-mount to the drive pool with the datasets that are NFS-shared (same as SMB-shared ones).
  • Apps that need to tap into data, use NFS shares (such as Jellyfin, Immich, Frigate) provided by this CT.
• Two VMs are of Debian, running Docker for my apps.
• VMs and CTs are all stored on 2x2TB M.2 NVMe SSDs.
• Data is stored in folders (per the NFS/SMB shares) on a 4x8TB ZFS pool with specific datasets like Media, Documents, etc. and a 1x4TB SSD ZFS "pool" for Frigate camera footage storage.

Due to having hardware passed-through to the VMs (GPU and Google Coral TPU) and using hardware resource mappings (one node as an Nvidia RTX A2000, two have Nvidia RTX 3050s - can have them all with the same mapped resource node ID to pass-through without issue despite being different GPUs), I don't have instant HA failover.

Additionally, as I am using ZFS with data on all three separate nodes, I understand that I have a "gap" window in the event of HA where the data on one of the other nodes may not be all the way up-to-date if a failover occurs before a replication.

So after all the above - this brings me to my question - what is the best way to replicate data that is not a VM or a CT, but raw data stored on those ZFS pools for the SMB/NFS shares - from one node to another?

I have been using Sanoid/Syncoid installed on one node itself, with cronjobs. I'm sure I'm not using it perfectly (boy did I have a "fun" time with the config files), and I did have a headache with retention and getting rid of ZFS snapshots in a timely manner to not fill up the drives needlessly - but it seems to be working.

I just setup the third node (the "analogous" one in specs) which I want to be the active "primary" node and need to copy data over from the other current primary node - I just want to do it intelligently, and then have this node, in its new primary node role, take over the replication of data to the other two nodes.

 

[pr0927]

Sorry to bump, not sure if no response on this because I presented my question in some stupid or unwelcome fashion, or if my question has somehow perplexed many (unlikely!) - but still hoping for some insight from those smarter than me.

 

[UdoB]

no response on this

There are just too many possible answers ;-)

I am old, so I would either opt for classic and simple "rsync" or ZFS snapshot replication. There are several tools available for that - and to obtain some experience with it is completely up to you

Have fun!