What is the method to utilize efficient ZFS daily snapshots to keep a backup of containers and vms?

[verulian]

I would like to keep ZFS-based backups / snapshots in daily (1 month weekly), weekly (5 weeks), and monthly (6 months) intervals on servers so as to be able to readily rollback if need arises.

How is this accomplished in an automated fashion so as to both accomplish this and purge older snapshots on Proxmox via ZFS?

 

[Folke]

Hi,
currently, there is no native way of creating regular zfs snapshots via proxmox.
However, there are some workarounds:
You could use the API to create and prune native proxmox snapshots[1]. This would be a bit more involved, but you could still manage the snapshots via the UI. Another method would be to utilize something like zfs-auto-snapshot, this only captures the filesystem though, RAM and VM settings would not be recorded, and a rollback would need to be done via the cli

[1] https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/snapshot
[2] https://manpages.ubuntu.com/manpages/focal/en/man8/zfs-auto-snapshot.8.html

 

[verulian]

Thank you @Folke. Hmm, I am concerned about implementing a solution that will be brittle with updates and/or future plans of Proxmox developers. Is there a chance we can get some insight as to both future plans and what course of action here would be least brittle and most likely to perpetuate or be convertible into a future solution that is likely to be implemented? Obviously ZFS is nearly perfect for this kind of situation, but the tooling simply hasn't been implemented as far as the Proxmox/PVE administrative environment goes...

 

[Dunuin]

Also keep in mind that snapshots aren't backups. You still need proper backups in addition to it. And snapshots will grow in size the older they get. So usually you only make use of snapshots to easily rollback the last few hours or days. For everything above it would be better to do proper backups using something like a PBS which will also save you space and offers features like live restores, were you are able to use a VM while it is still being restored.

 

[verulian]

@Dunuin of course. But ZFS facilitates an easy extension of that as well through its facilities, but one step at a time. The first goal is to get this going so that it is relatively easy to roll back changes and other things (such as, heaven forbid, having a server hacked through some vulnerability and needing to roll back to a point prior thereunto). So snapshots are essential to have in these timeframes / this window that I noted above in my own experience since a host of scenarios can arise that necessitate this. So no, I'm not interested in the backup aspect of this - yet. Yes, Proxmox Backup Server (aka PBS) can be used for this other scenario, but it should also receive its data via ZFS as well if possible if we start to go down the deeper backup discussion.

 

[Dunuin]

The first goal is to get this going so that it is relatively easy to roll back changes and other things (such as, heaven forbid, having a server hacked through some vulnerability and needing to roll back to a point prior thereunto).

Thats one of the situations where you need proper ransomware-protected/immutable backups on another non-PVE server (have a look at PBS) and not snapshots, because when your PVE node gets hacked the attacker could destroy the whole pool (of all nodes when using replication / HA) including all the snapshots.

 

[verulian]

Thats one of the situations where you need proper ransomware-protected/immutable backups on another non-PVE server (have a look at PBS) and not snapshots, because when your PVE node gets hacked the attacker could destroy the whole pool (of all nodes when using replication / HA) including all the snapshots.

I understand the concern that snapshots alone are not sufficient for robust backup and ransomware protection. However, I believe a multi-layered approach leveraging ZFS's capabilities is optimal:

1. Implement regular automated ZFS snapshots on the Proxmox cluster itself for quick rollback and recovery from issues like misconfigurations, software bugs, user error, etc. Tools like zfs-auto-snapshot can enable this.
2. Utilize ZFS send/receive functionality (https://docs.oracle.com/cd/E18752_01/html/819-5461/gbchx.html) to continuously replicate filesystem-level snapshots from the Proxmox cluster to a separate Proxmox Backup Server (PBS). This provides an additional layer of backup that is independent from the primary system.
3. Configure PBS to make its received snapshots immutable for a defined time period. This protects the backup from being altered/deleted, even if the primary system is compromised.
4. If desired, supplement the ZFS snapshot backups with regular Proxmox-native VM/container backups to PBS. This captures additional VM/container metadata and enables full VM recovery. It is possible here as well just to backup only the metadata so that restoration with such can be performed in conjunction with the use of the snapshots for more storage efficiency.

So in summary, my suggestion is to use ZFS snapshots for fast, space-efficient recovery on the primary system, while also replicating those snapshots to an external immutable backup server for mistake, testing, playing, ransomware protection, disaster recovery, etc. And perhaps layering or integrating some fashion of Proxmox-native backups on top of this provides further coverage.

Such a multi-faceted strategy leveraging ZFS and PBS in tandem is a robust approach that mitigates concerns around relying solely on snapshots, while still gaining ZFS snapshot benefits.

The primary use of snapshots, to reiterate, is not just about backup/disaster recovery primarily, but for robust and storage-efficient rollbacks and integrating all use cases.

 

[Dunuin]

The primary use of snapshots, to reiterate, is not just about backup/disaster recovery primarily, but for robust and storage-efficient rollbacks and integrating all use cases.

Agree. More backups are always better unless you are running out of budget. And snapshots for "issues like misconfigurations, software bugs, user error, etc" in addition to backups are a good idea. I'm just confused that you want to store the snapshots for so long. 6 months is a lot of time for reverting misconfigurations. And keeping snapshots for 6 months means space of data you delete/edit will only be freed up after 6 months. Depending on how fast your data changes, how long you store them and how granular you keep them, the snapshots could consume a multiple of the size of the actual data on the virtual disks or your backups of them on your PBS.

 

[fmaione]

Hi,
currently, there is no native way of creating regular zfs snapshots via proxmox.
However, there are some workarounds:
You could use the API to create and prune native proxmox snapshots[1]. This would be a bit more involved, but you could still manage the snapshots via the UI. Another method would be to utilize something like zfs-auto-snapshot, this only captures the filesystem though, RAM and VM settings would not be recorded, and a rollback would need to be done via the cli

[1] https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/qemu/{vmid}/snapshot
[2] https://manpages.ubuntu.com/manpages/focal/en/man8/zfs-auto-snapshot.8.html

Thanks for the [1] tip, it opened a whole world to me. For the [2], if I have PVE on ZFS (so I have the rpool) zfs-auto-snapshot would backup also the vmid.conf, so settings will be preserved. Am I wrong? Am I missing something?