Ever since Proxmox 6 came out, we moved towards using Proxmox with ZFS across all physical servers. That is because with the new corosync, we don't need to have multicast traffic between those servers. We have successfully virtualized servers that were previously on bare metal and even started using more exotic solutions like ZFS inside a KVM guest that is on a ZFS zvol.
We this setup, we replicate almost every container or VM in the cluster according to a schedule (configured in Proxmox GUI) to a hot-spare AND a backup server (SATA disks with lots of storage). The intention is that there is a replicated host-spare that can run AND a copy on a third server across DCs that will be persisted.
We previously used pve-zsync but found the options to be very limited - for example, when a container is migrated in a cluster, the PVE server running pve-zsync in pull mode can no longer find it because the IP address of the source is outdated.
We need to take snapshots and persist them over time - something that is built into Proxmox in the form of scheduled backups. Except for ZFS - those make no sense at all with ZFS as they don't use snapshots but archive the whole dataset/zvol and waste tons of space. On the other hand, replication has no way to keep more than the latest snapshot (it's designed to just copy current state and keep a snapshot to be able to resume replication).
We've tried using two solutions designed for scheduling ZFS snapshots: https://github.com/oetiker/znapzend/ and https://github.com/jimsalterjrs/sanoid.
Since we are already replicating the VMs to the backup node, there is no need to send/receive the ZFS datasets - we let Proxmox handle it. The only remaining think to do is to take snapshots and rotate them. Both of these tools failed: ZnapZend stores configuration in the ZFS dataset properties, which on a replication target means they are being overwritten; while snapshots created by Sanoid are simple removed by the next PVE-executed replication.
A simple solution would be to limit PVE's handling of snapshots to simply the ones that it actually created, so as not to drop the snapshots creating externally. The only working workaround is to duplicate the dataset locally via send/receive, which also duplicated the amount of space used and is therefore wasteful.
In the process of orchestrating the snapshots, we also came across an issue of how the quota for containers is set for the datasets - it makes the snapshots count, which makes sense if VM Owner can initiate those but does not, when the burden of snapshots is to be taken on by the provider. This issue is documented here: https://bugzilla.proxmox.com/show_bug.cgi?id=2201
Other bugs related to this thread:
* https://bugzilla.proxmox.com/show_bug.cgi?id=2063
* https://bugzilla.proxmox.com/show_bug.cgi?id=2022
TL;DR: We need ZFS snapshots on schedule in a Backup functionality. In addition, the ability to offload snapshots to select pvesr replication targets.
We this setup, we replicate almost every container or VM in the cluster according to a schedule (configured in Proxmox GUI) to a hot-spare AND a backup server (SATA disks with lots of storage). The intention is that there is a replicated host-spare that can run AND a copy on a third server across DCs that will be persisted.
We previously used pve-zsync but found the options to be very limited - for example, when a container is migrated in a cluster, the PVE server running pve-zsync in pull mode can no longer find it because the IP address of the source is outdated.
We need to take snapshots and persist them over time - something that is built into Proxmox in the form of scheduled backups. Except for ZFS - those make no sense at all with ZFS as they don't use snapshots but archive the whole dataset/zvol and waste tons of space. On the other hand, replication has no way to keep more than the latest snapshot (it's designed to just copy current state and keep a snapshot to be able to resume replication).
We've tried using two solutions designed for scheduling ZFS snapshots: https://github.com/oetiker/znapzend/ and https://github.com/jimsalterjrs/sanoid.
Since we are already replicating the VMs to the backup node, there is no need to send/receive the ZFS datasets - we let Proxmox handle it. The only remaining think to do is to take snapshots and rotate them. Both of these tools failed: ZnapZend stores configuration in the ZFS dataset properties, which on a replication target means they are being overwritten; while snapshots created by Sanoid are simple removed by the next PVE-executed replication.
A simple solution would be to limit PVE's handling of snapshots to simply the ones that it actually created, so as not to drop the snapshots creating externally. The only working workaround is to duplicate the dataset locally via send/receive, which also duplicated the amount of space used and is therefore wasteful.
In the process of orchestrating the snapshots, we also came across an issue of how the quota for containers is set for the datasets - it makes the snapshots count, which makes sense if VM Owner can initiate those but does not, when the burden of snapshots is to be taken on by the provider. This issue is documented here: https://bugzilla.proxmox.com/show_bug.cgi?id=2201
Other bugs related to this thread:
* https://bugzilla.proxmox.com/show_bug.cgi?id=2063
* https://bugzilla.proxmox.com/show_bug.cgi?id=2022
TL;DR: We need ZFS snapshots on schedule in a Backup functionality. In addition, the ability to offload snapshots to select pvesr replication targets.
