Operational issues with ZFS and pvesr

Don Daniello

Active Member
Jan 28, 2012
60
5
28
Ever since Proxmox 6 came out, we moved towards using Proxmox with ZFS across all physical servers. That is because with the new corosync, we don't need to have multicast traffic between those servers. We have successfully virtualized servers that were previously on bare metal and even started using more exotic solutions like ZFS inside a KVM guest that is on a ZFS zvol.

We this setup, we replicate almost every container or VM in the cluster according to a schedule (configured in Proxmox GUI) to a hot-spare AND a backup server (SATA disks with lots of storage). The intention is that there is a replicated host-spare that can run AND a copy on a third server across DCs that will be persisted.

We previously used pve-zsync but found the options to be very limited - for example, when a container is migrated in a cluster, the PVE server running pve-zsync in pull mode can no longer find it because the IP address of the source is outdated.

We need to take snapshots and persist them over time - something that is built into Proxmox in the form of scheduled backups. Except for ZFS - those make no sense at all with ZFS as they don't use snapshots but archive the whole dataset/zvol and waste tons of space. On the other hand, replication has no way to keep more than the latest snapshot (it's designed to just copy current state and keep a snapshot to be able to resume replication).

We've tried using two solutions designed for scheduling ZFS snapshots: https://github.com/oetiker/znapzend/ and https://github.com/jimsalterjrs/sanoid.
Since we are already replicating the VMs to the backup node, there is no need to send/receive the ZFS datasets - we let Proxmox handle it. The only remaining think to do is to take snapshots and rotate them. Both of these tools failed: ZnapZend stores configuration in the ZFS dataset properties, which on a replication target means they are being overwritten; while snapshots created by Sanoid are simple removed by the next PVE-executed replication.

A simple solution would be to limit PVE's handling of snapshots to simply the ones that it actually created, so as not to drop the snapshots creating externally. The only working workaround is to duplicate the dataset locally via send/receive, which also duplicated the amount of space used and is therefore wasteful.

In the process of orchestrating the snapshots, we also came across an issue of how the quota for containers is set for the datasets - it makes the snapshots count, which makes sense if VM Owner can initiate those but does not, when the burden of snapshots is to be taken on by the provider. This issue is documented here: https://bugzilla.proxmox.com/show_bug.cgi?id=2201

Other bugs related to this thread:
* https://bugzilla.proxmox.com/show_bug.cgi?id=2063
* https://bugzilla.proxmox.com/show_bug.cgi?id=2022

TL;DR: We need ZFS snapshots on schedule in a Backup functionality. In addition, the ability to offload snapshots to select pvesr replication targets.
 
  • Like
Reactions: guletz and jmcorrea
Both of these tools failed: ZnapZend stores configuration in the ZFS dataset properties, which on a replication target means they are being overwritten

Could you explain? I also have the problem and wanted to use zfs-zed, znapzend oder Sanoid.

The ability to have multiple snapsots with generation schedules like znapzend ist crucial. Almost everything is there, but unfortunately, it seems is not compatible currently with proxmox.
 
HI Don,

You do not want to run znapzend and PVE storage replication between the same nodes. They do conflict and most likely will cause PVE storage replication to fail at some point.

We are, however, successfully running a combination of local PVE ZFS replication and znapzend for off-site (remote) replication.
Local replication replicates everything, including all local snapshots, but only within a cluster.

The nice thing is that znapzend can differentiate between local and remote retension policies. We are using this to keep snapshots longer on the remote (=ultimate) backup site and only the last few days on the local sites (as they are using SSDs, making retention more expensive).

We run znapzend at full hours and, for critical VMs/CTs run replication at 15/30/45 minutes every hour.

I have created a bunch of scripts to cope with znapzend configurations and VM/LXC migration between PVE nodes. These ensure that znapzend configuration is „migrated“ along - which is important. Happy to share if you need.
 
[QUOTE="rholighaus, post: 278903, member: 41611"
I have created a bunch of scripts to cope with znapzend configurations and VM/LXC migration between PVE nodes. These ensure that znapzend configuration is „migrated“ along - which is important. Happy to share if you need.
[/QUOTE]

I also experimented with znapzend, but I had problems. I wanted to have short lived 15min local backups and replicate them every 12 hours to remote site where there are stored longer, but znapzend was replicating the short lived snapshots also.

I would be interessted in you scripts to.

It would be a good thing, if proxmox had the possibility to keep the cluster intnal replication snapshots for a configurable time to have them serve also a a versioned backup. Even better would be, if the GUI could display these snaphots and let the user revert to one of them or clone one of them to get old data back. For Windows VMs it would also be nice, if proxmox could clone the VM disk for the snaphot and mount it a NFTS, so there would be an immeate acces to old files without going to the command line.
 
  • Like
Reactions: guletz

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!