[TESTERS NEEDED] Testers welcome: zpaqfranz backups on real Proxmox setups

[fcorbelli]

Hello everyone,
First of all, thank you for stopping by.

Just to be clear: this thread is not an advertisement, a comparison, or any criticism of vzdump and Proxmox Backup Server.
They are phenomenal, industry-standard tools built by people who know this platform far better than I do.
If you are happy with your backups, you are doing it right.
If fact... I use vzdump too

I am the author of zpaqfranz, an open-source (MIT) archiver.
This means, BTW, that I gain no money, then you can continue reading

I recently added support for Proxmox guests, but my server farm is small.
What truly counts is how it behaves on diverse, real-world setups.
Therefore, I am kindly asking (only if you are curious and have spare time) for help testing it.
Any feedback (especially "this is wrong, and here is why") is immensely welcome.

Please note: This code is still marked as not ready for production. Run it alongside your usual backups, never in place of them, and start with a small, disposable guest.

What is zpaqfranz?
Something like 7z on steroid

• Standalone: No server, no database, no repository configuration needed.
• Append-only archive: Easy to copy with rsync, rclone, etc.
• Deduplication: Each run creates a new version; only new data takes up space.
• Verifiable: Fully hashed, allowing you to test integrity anytime. Encryption is optional.

Project pages: https://github.com/fcorbelli/zpaqfranz
https://sourceforge.net/projects/zpaqfranz/

The key element (on standard vzdump) is the deduplicator, aka can save multiple files just like snapshots.

Quick Installation

You need version 65.17 or later.
Here for example https://github.com/fcorbelli/zpaqfranz/releases

Built for any recent Debian/PVE (tested on PVE 9.2), with something like:

wget https://github.com/fcorbelli/zpaqfranz/releases/latest/download/zpaqfranz_oracle79 -O /usr/local/bin/zpaqfranz
chmod +x /usr/local/bin/zpaqfranz
zpaqfranz autotest

(Or (even better) build it yourself from zpaqfranz.cpp with g++ -O3 -Dunix zpaqfranz.cpp -o zpaqfranz -pthread -ldl)

Two Backup Approaches

1. proxvzdump (The "Universal" Way)
It runs vzdump and archives the stream WITH DEDUPLICATION

• Pros: Works with virtually any storage supported by vzdump; safe and conservative.
• Cons: Reads the whole guest every time; standard vzdump streams are never 100% byte-identical, so a no-change run still consumes a little space (MBs).

How?

mkdir -p /backup
zpaqfranz proxvzdump /backup 100                    # Single guest (VM/CT), one in each file
zpaqfranz proxvzdump /backup -all -not 300            # All guests except ID 300
zpaqfranz proxvzdump /backup/all.zpaq -all           # All guests in one deduped file
zpaqfranz proxvzdump /backup -all -key mypassword    # Encrypted

2. zfsproxbackup (The "Targeted" Way)
No vzdump: takes a storage snapshot and reads the disks directly

• Pros: Unchanged data costs KB instead of MB; container rootfs stored file-by-file (allows single-file restores); disks/files can be extracted independently of Proxmox.
• Cons: Limited to tested storages (ZFS, LVM-thin, btrfs, folders); unsupported storages are rejected. Without a guest agent, running VMs are backed up crash-consistent.

zpaqfranz zfsproxbackup /backup 100     # VM on local-zfs (snapshot + clone, running)
zpaqfranz zfsproxbackup /backup 200     # Container on local-zfs (file-by-file, running)
zpaqfranz zfsproxbackup /backup 103     # VM/CT on LVM-thin or btrfs
zpaqfranz zfsproxbackup /backup -all    # Whole node (skips unsupported automatically)

Checking & Restoring

Check integrity and list versions:

zpaqfranz t /backup/proxvzdump-qemu-100.zpaq
zpaqfranz i /backup/proxvzdump-qemu-100.zpaq

Extract a single file from a container backup:

zpaqfranz x /backup/zfsproxbackup-lxc-200.zpaq lxc-200/rootfs/etc/hostname -to /tmp/out/

Restores: Please do not test restores on critical data. If you test proxvzrestore or zfsproxrestore, always target a new VM ID without -force. Restoring directly to ZFS streams (zfsreceive) is currently recommended only for very experienced admins on test nodes.

Where Your Help is Most Needed

Real-world scenarios are invaluable:

• Clusters, PVE 7/8, Ceph (RBD/CephFS), NFS, CIFS, iSCSI, ZFS-over-iSCSI...
• Encrypted ZFS datasets, UEFI/TPM disks...
• Heavy-load VMs with guest agents active...
• Containers with mount points/bind mounts...

What to Report (Here or on GitHub)
If something goes wrong (or even if it just works!), sharing these details helps immensely:

pveversion
zpaqfranz | head -1
# Storage type, VM/CT, running status, approx size
# Exact command used
echo $?                      # (0 OK, 1 warning, 2 error)
# Last 20 lines of output & archive size growth

Issues: https://github.com/fcorbelli/zpaqfranz/issues

---

Thank you very much for reading, and a heartfelt thanks to anyone willing to test this out.
Your setups make all the difference, and a special thanks to the Proxmox team and community for an incredible platform!

Kind regards,
Franco

PS And yes, there is a (kind of) '80 console too

zpaqfranz work proxmox mygood.proxmox.server

 

[fcorbelli]

A quick add-on: why build zpaqfranz around vzdump?

To keep the first post readable, I left out the main "why" behind the proxvzdump approach.

The core idea is simple: keep long-term historical backups of vzdump streams without running out of space or bandwidth

Standard vzdump is reliable, but it has a few practical challenges when storing many versions over time:

• No built-in encryption: A potential issue if you offload backups to remote or untrusted storage.
• Stream variance: Even if guest data changes very little, consecutive vzdump runs generate different stream outputs, forcing full transfers over rsync.
• Pruning needed: You eventually (sooner or later) have to delete old dumps to free up space.

Wrapping vzdump inside zpaqfranz addresses these points:
1. Strong Encryption: You can apply native AES encryption, FRANZEN (a block cipher ChaCha-derivative optimized for modern SSD storage), or even both sequentially.
2. Append-only & Network-friendly: Archives grow strictly at the end. You can push updates using rsync --append (or directly via zpaqfranz's built-in SFTP support if compiled with libcurl), transferring only the newly added data. Split/multipart archives with fixed max file sizes are also supported.
3. Deep Versioning: You can keep dozens or hundreds of daily backups like a time machine without manual pruning (pruning commands exist, but are rarely needed).

---

A Practical Example

Imagine backing up a VM with a 100 GB disk that barely changes day-to-day:

Day 1:

• You run vzdump =>Output is ~100 GB.
• zpaqfranz compresses/deduplicates it into an archive => Size on disk: ~100 GB (just an example!)
• First transfer to remote storage: 100 GB sent.

Day 2 (only 1 GB of real changes inside the guest):

• Without zpaqfranz: A new vzdump generates another ~100 GB file. Total storage: 200 GB. Total network transfer: again ~100 GB.
• With zpaqfranz: You run zpaqfranz proxvzdump targeting the same archive.
• Storage result: The archive grows by ~1 GB (Total: 101 GB for both versions).
• Network transfer: Running rsync --append sends only the new 1 GB.

You now have two full restorable restore points (aka two offline-encrypted "snapshots") using 101 GB of space instead of 200 GB, and sending 1 GB over the wire instead of 100 GB.

A Few Caveats
This is technology, not magic!

• If guest disk contents undergo major churn (e.g. database re-indexing, heavy video processing, or full-disk OS upgrades), archive growth will match the actual changes (of course)
• If the guest OS uses full-disk encryption internally, deduplication efficiency will drop significantly (of course)

---

A Note on Credits:
While zpaqfranz is my own fork (actively maintained), the core credit for the underlying zpaq format and deduplication engine belongs entirely to Dr. Matt Mahoney, creator of the original zpaq project.

 

[fcorbelli]

Part 3: Why process ZFS streams with zpaqfranz?

After discussing vzdump, a natural question arises: "If I already use ZFS, why non just use native ZFS send/receive?"

ZFS snapshots and streams are a fantastic technology. They offer blazing-fast incremental backups with negligible performance impact because you don't need to re-read the entire disk.

However, managing raw ZFS stream files manually comes with real-world headaches:

• Chain dependency risk: Incremental ZFS streams rely on an unbroken chain. If you lose a single intermediate stream file, the entire rollback path after it can become unrestorable.
• No built-in encryption on wire/destination: Unless you are sending to an encrypted ZFS dataset, raw stream files sitting on a destination server are unencrypted.
• Management overhead: Handling dozens of daily `.zfs` stream files across remote storages or cloud backends quickly becomes messy.

How zpaqfranz helps with ZFS streams
When using zfsproxbackup, zpaqfranz acts as a single, self-contained repository for your ZFS data:

• Single-file repository: Instead of managing hundreds of individual stream chunks, everything goes into one append-only archive.
• Safety & Integrity: Every block is hashed. You can test archive integrity at any moment to ensure no "missing pieces" exist before you actually need to restore.
• Encryption & Transport: Streams get encrypted on-the-fly (AES/FRANZEN) before written, and can be efficiently pushed via rsync --append or SFTP.

---
My Recommended Strategy: The Hybrid Approach
In my own routine, I actually use both approaches to get the best of both worlds:

1. Intraday backups via ZFS (zfsproxbackup):
Run every few hours during the working day. Extremely fast, negligible I/O impact on live guests, minimal space consumed.
2. Nightly full backup via vzdump (proxvzdump):
Run once at night when the node is quiet. It provides a conservative, standalone "fail-safe" archive that doesn't depend on ZFS features.

And, of course, an remote-site Proxmox Backup too