Realtime ZFS Replication with the new cs-sync
Part of the napp-it 4ai web-gui for any OS but it is Opensource and runs as a simple cli tool
https://www.napp-it.org/cs-tools.htmlZFS replication normally works asynchronously and job-based. On a time grid, for example every hour, a snapshot is created and sent with the changed data blocks to a local or remote target ZFS filesystem to keep both in sync.
If you want realtime ZFS replication, for example for disaster recovery, you have to work event-driven. A write to a monitored ZFS dataset triggers the replication immediately. Because the data to send is always very small and is transferred very quickly, the delay between a change on the source and the change on the destination filesystem is minimal. What is needed after the initial replication is sensible management of the snapshot pairs, sensible timing on heavily loaded filesystems, and a resume after a network failure or a reboot, for example of the backup server. Realtime ZFS replication is especially useful for VM storage such as Proxmox, where file-based sync quickly reaches its limits because of the large files.
With Proxmox there are two more aspects. First, the VM configuration is not stored in the dataset as with ESXi, but in /etc/pve on Proxmox. To restore the realtime-replicated dataset of a VM, the VM configuration is required. When cs-sync finds at start that the source is a Proxmox host, it reads in every cycle the configuration file of each VM or container whose disk datasets are part of the sync (not the whole /etc/pve), and attaches it as the ZFS user property cs
veconf to the associated dataset and to every sync snapshot. The property is transferred to the backup by the cs-sync protocol (not inside the zfs send stream); a pure configuration change goes over without a new send.Second, a snapshot in Proxmox is not consistent in every case but has to be seen like a power failure, harmless to delicate depending on the VM. On a Proxmox source, cs-sync therefore sends the first snapshot of each day automatically through cs-freeze4snap, which freezes the VMs of the source dataset before the snapshot (guest agent, container freeze). This snapshot is kept as kf<MMDD> on both sides and marked with cs:consistent=app, or with paused if a VM could only be paused (then it is still only crash-consistent). If cs-sync does not find cs-freeze4snap (it looks in the same folder as cs-sync, in the napp-it tools folder and in PATH), the freeze is skipped with a warning in the log. On days without data changes no new snapshot is created. So there is at least one reliably restorable state per day.
The existing cs-sync stays
Folder Primary <-> Secondary: event-driven two-way sync, for example syncing an SMB share with an NFS/S3 share
Folder Primary -> local or remote backup: realtime sync, file-based or now also via ZFS send.
New: manuals for all cs-tools
https://www.napp-it.org/cs-tools.html
