PBS's verify job is good at what it does: it proves the backup's data is intact. What it can't tell you is whether the guest will actually boot, mount its filesystem read-write and bring its services up. I wanted that proof on a schedule, the way VMware Live Recovery's test recovery or Xen Orchestra's backup verification gave it to people, so I built PBP (PVE Backup Proofing).
What it does, on a schedule, for each guest:
- takes the newest backup from PBS (guests whose last proof is oldest go first) and protects it from pruning while it works
- restores it as a temporary guest in a VMID range you reserve, with every NIC moved onto an isolated bridge, start-on-boot off and passthrough removed
- boots it and checks inside: failed systemd units, a read-write root, service profiles (PostgreSQL, MariaDB, nginx, Docker, Redis, SSH and so on), Windows automatic services through the guest agent, OpenRC on Alpine and Gentoo, plus your own per-guest checks
- destroys the copy, puts the backup's protection back, writes a verdict into the backup's notes, and sends one summary through your existing Proxmox notification targets
Safety around production was the part I spent the most time on:
- it never logs in as root: a narrow API token, and SSH only as an unprivileged pbp user locked to one helper command
- it checks that the proof bridge really is isolated before it uses it
- a janitor cleans up after a crashed run, and capacity is checked before anything is restored
- releases are signed
Requirements: Proxmox VE 9 and PBS 4. PBP itself runs in Docker or from a .deb (amd64 or arm64), ideally on a separate host. **In a test or lab environment, it's fine on the same host, in a nested docker host, etc., but like everything else I recommend not putting this on the PVE host itself....**
Real result from my own lab: a scheduled run restored and booted a container's newest backup and found its gitea-runner.service had failed. The backup was "verified" in PBS, and I'd only have found out on the day I needed it.
Guest with failed service, would not have been caught until reboot or restore of the guest It's free and open source (Apache-2.0), currently v0.8.2, so still pre-1.0.
- GitHub: https://github.com/Breadth-Security/PVE-Backup-Proofing
- Overview and install steps: https://breadthsecurity.com
- Does this fill a gap in how you handle backups today, or do you already prove restores another way?
- Which checks would you want that aren't there?
- Anything in the setup or the safety model that would stop you running it near production?
- Any edge cases (OS's or otherwise). I have a pretty good collection of various services and operating systems running that I have acquired over the years, but I am sure that I am missing something that you all use frequently.
Thanks all!
