REPLICATION is ZFS only? Why? And what is PVE-zsync?

[esi_y]

When I look at pvesr [1] source [2] it uses PVE::API2::Replication [3] which gets to use PVE::Replication [4] which in the end just calls PVE::Storage::storage_migrate [5] which essentially does pvesm import [6] on the other end and feeds it with pvesm export [7].

Unless I looked wrong, what's the problem supporting replication on any filesystem that has snapshots? There's no zfs send|receive at play.

Also, what is the point pve-zsync (alongside the said replication API), I read the docs [8] but could not quite get the rationale for both co-existing.

[1] https://pve.proxmox.com/pve-docs/pvesr.1.html
[2] https://github.com/proxmox/pve-manager/blob/master/PVE/CLI/pvesr.pm
[3] https://github.com/proxmox/pve-mana...f0590ddeb58fab1ad/PVE/API2/Replication.pm#L39
[4] https://github.com/proxmox/pve-gues...9b3172296c38058d0/src/PVE/Replication.pm#L220
[5] https://github.com/proxmox/pve-stor...3e84d0a18f95de0322168/src/PVE/Storage.pm#L702
[6] https://github.com/proxmox/pve-stor...3e84d0a18f95de0322168/src/PVE/Storage.pm#L820
[7] https://github.com/proxmox/pve-stor...3e84d0a18f95de0322168/src/PVE/Storage.pm#L743
[8] https://pve.proxmox.com/wiki/PVE-zsync#PVE_Storage_Replication_and_PVE-zsync

 

[shanreich]

pvesm import | export do in fact use zfs send | recv [1] [2]

Unless I looked wrong, what's the problem supporting replication on any filesystem that has snapshots?

In theory, there is nothing that would prevent supporting replication for other storage types that support similar commands. It simply has not been implemented for other storage types (yet).

[1] https://git.proxmox.com/?p=pve-stor...a2864a2368bf3c3ba436ddca8047a4bf;hb=HEAD#l474
[2] https://git.proxmox.com/?p=pve-stor...2b62df7ebeda572cdf56ef19be872d56;hb=HEAD#l777

 

[sterzy]

Except the volume_import [1] and volume_export [2] functions, use zfs send/receive.

If you look at the other storage plugins, for example LVM Thin (which does support snapshots), there is no volume_export function. As for BTRFS, which would support something similar, it is still considered a technological preview.

Also, pve-zsync, that's a utility that does some things differently from replication. For example, you can sync snapshots to a storage outside your cluster, and its main purpose is closer to backing up data than replication, which itself is useful for HA purposes. It also can't handle migrating VMs properly for example.

Also note that such functionality doesn't really make sense for distributed storage. RBD supports snapshots, but replicating within the same cluster wouldn't really make sense there. Ceph has RBD Mirroring [3], though, if you need to mirror an RBD image across several Ceph clusters.

[1]: https://git.proxmox.com/?p=pve-stor...PVE/Storage/ZFSPoolPlugin.pm;h=3669fe152#l777
[2]: https://git.proxmox.com/?p=pve-stor...PVE/Storage/ZFSPoolPlugin.pm;h=3669fe152#l736
[3]: https://pve.proxmox.com/wiki/Ceph_RBD_Mirroring

 

[esi_y]

pvesm import | export do in fact use zfs send | recv [1] [2]

Sloppy me! I did not look there (anymore) assuming wrongly it's doing it its own way - there was another thread I was following yesterday when OP was using manually zfs send|receive across nodes but complained he could not start such VM. I guess that's down to dataset/snapshot naming system then.

In theory, there is nothing that would prevent supporting replication for other storage types that support similar commands. It simply has not been implemented for other storage types (yet).

Thanks!

 

[esi_y]

Except the volume_import [1] and volume_export [2] functions, use zfs send/receive.

If you look at the other storage plugins, for example LVM Thin (which does support snapshots), there is no volume_export function. As for BTRFS, which would support something similar, it is still considered a technological preview.

You got me, I asked because I was eying the add it for BTRFS!

Also, pve-zsync, that's a utility that does some things differently from replication. For example, you can sync snapshots to a storage outside your cluster,

I figured, unless it migrated away. It feels like pve-zsync was preview of what then became Replication but not complete replacement.

and its main purpose is closer to backing up data than replication, which itself is useful for HA purposes. It also can't handle migrating VMs properly for example.

This is something I would like to know actually, why. I mean I can go read what it's doing but since you already replied ...

Also note that such functionality doesn't really make sense for distributed storage. RBD supports snapshots, but replicating within the same cluster wouldn't really make sense there. Ceph has RBD Mirroring [4], though, if you need to mirror an RBD image across several Ceph clusters.

No worries, just for small clusters replication is much better than even eyeing CEPH.

 

[sterzy]

This is something I would like to know actually, why. I mean I can go read what it's doing but since you already replied ...

Well the short answer: Historical reasons, probably.

The long answer: Replication was introduced with a no longer supported storage backend called DRBD (not to be confused with RBD). pve-zsync was first released as a tech-preview for PVE 3.4, which was also the first release to support ZFS [2]. It was, and is, basically a standalone package that can handle syncing ZFS snapshots, in a way it's just a Perl script that wraps zfs send|recv, ssh and cron and not much else.

As it was intended to be separate of Proxmox VE, it couldn't deeply integrate with it and, hence, it wasn't suitable for HA purposes. As an aside, note that replication still isn't ideal, as you can lose data in every replication scenario. Proxmox VE 4.4 removed DRBD support and replaced it with an external plugin provided by Linbit [4]. Finally, with Proxmox VE 5 we get to ZFS replication [5], which more deeply integrates with Proxmox VE, and, thus, could take the place of previous replication implementations.

Hope that makes sense.

No worries, just for small clusters replication is much better than even eyeing CEPH.

Yeah, in a way replication is what you do when you can't use a shared storage.

[1]: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_1.4_beta1
[2]: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_3.4
[3]: https://git.proxmox.com/?p=pve-zsync.git;a=blob;f=pve-zsync;h=98190b208
[4]: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_4.4
[5]: https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_5.0