PBS vs Veeam B&R Proxmox Plugin - QEMU live snapshot implementation & shared LVM storage

[alma21]

Hi,

haven't found any comprehensive info/docs how the Veeam Proxmox plugin is integrated with PVE, but as far as understand:
(please correct/extend if I'm wrong)

high level procedure (full backup only, excluded CBT/dirty bitmaps for incrementals):

1) a Veeam worker VM is started on the same PVE node where the VM/disk(s) resides which should be backuped (if so configured & Hot-Add should be used - otherwise NBD/NW mode only)
2) the QEMU live snapshot function is called via PVE/Qemu API - similiar to PBS (but only initially*) - to ensure that also snapshot creation is possible on non snapshotable storage -> like raw (shared) LVM
3) with the beginning of the backup job the VM is shortly freezed and the base VM disk(s) is put into readonly and are then attached to the worker VM to get a consistent disk image for transfer to the backup server
4) an overlay qcow2 image is created on a previous defined file level storage/directory - to hold new writes for the VM(s) during the backup procedure
5) when the backup process has finished the hot-add vm disk(s) is reattached to the original VM and a snapshot merging/flattening(?) of the qcow2 overlay happens back to the raw (?) image.

- *AFAIK PBS (without fleecing) puts some kind of COW filter into but does not create an additional overlay/qcow2 disk - so new VM writes during the backup are >always< written (with maybe some delay) to the original VM disk(s)/storage
also with a fleecing device - only old data is temporarly stored on an extra device/storage

- both solutions use --- with any (?) included/default PVE storage (plugins) --- the Qemu live snapshots (and no e.g. ZFS / LVM thin storage snapshots on corresponding storage)


I'm asking/writing this because:

- of course to get a better technical understanding how it works

- but specially the Veeam file level storage/directory for snapshots during backups (pt. 4.) makes me a little bit wondering - neither in the Veeam community forum nor in the off. docs it is clearly mentioned which (negative) impact this can have if not wisely selected/configured ... >>> pls use some file temp space which is needed for worker & backup snapshots ... <<<
e.g.: all VM disks in a PVE cluster with many nodes on shared LVM storage (on HA/high performance/$$$ SAN) -> then the suggestion is e.g. to use PVE local storage/one separate SSD disk/VM NFS etc.

maybe I'm wrong but:

- if this temp. (ev. low-grade/non redudant/small) backup & snapshot file storage fails during backup process - all new written VM data is lost
- if a slow local device or file share is used then the backup process(es) can ev. stumble with (heavy) VM writes - the high performance SAN doesn't help much
- if the qcow2 overlay gets merged back this puts additional stress to the VM(s) and storage


so imho esp. for a big PVE cluster & shared LVM storage solution I would use a LUN/LVM/FS on the SAN - dedicated to some of the nodes for this Veeam snapshot/worker file space

if all this is correct - then at least for PVE VM only backups - imho ... PBS looks like the better integrated and/or "less efforts" approach

 

[bbgeek17]

Hi @alma21

haven't found any comprehensive info/docs how the Veeam Proxmox plugin is integrated with PVE

Only Veeam can give you this level of technical details.

the QEMU live snapshot function is called via PVE/Qemu API

From our testing of their initial plugin release - they bypass the PVE completely and interact with QEMU directly.

- *AFAIK PBS (without fleecing) puts some kind of COW filter into but does not create an additional overlay/qcow2 disk - so new VM writes during the backup are >always< written (with maybe some delay) to the original VM disk(s)/storage

the filter is used with and without fleecing. Fleecing assist with in-line write requests, but the underlying mechanism is the same.

also with a fleecing device - only old data is temporarly stored on an extra device/storage

correct, the VM IO is no longer stalled until the out-of-order backup is complete, rather the block is placed in fleecing storage to avoid IO delays.

- both solutions use --- with any (?) included/default PVE storage (plugins) --- the Qemu live snapshots (and no e.g. ZFS / LVM thin storage snapshots on corresponding storage)

both solutions are completely storage independent and do not rely on any storage plugin specific functionality, in particular no snapshots are created by either. This is a common point of confusion for people coming from Veeam/VMware.

if all this is correct - then at least for PVE VM only backups - imho ... PBS looks like the better integrated and/or "less efforts" approach

Generally complimentary products from the same company are better integrated with each other. Whether that causes political/economical issues is a different question!

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[iceph03nix]

We currently use both as we already had Veeam for ESXI, and it has some features that PBS does not support (yet?) Mostly in the Application Aware space.

I really like the PBS backup process and it's simplicity in getting set up. I especially like that there's not really a distinction between a full backup and a diff backup. I don't have to worry about making sure that my active fulls are aligned with the retention policy or making a synthetic full.

Where I find PBS lacking is in backing up things like SQL DBs as well as Veeam does. Veeam will interact with the DB itself, truncate logs, and you can do state restores of the DB without having to mess with the full VM.

Also, the Veeam file restore options are more robust. Being able to tell it to just put files back how they were is pretty awesome. on PBS, from what I've seen, you have to download the archive, and then restore it manually. With PVE, it does require the deployment of their agents for restore, which we didn't have to mess with on ESXI, but that's pretty minimal and can be done when needed.

It's possible I'm ignorant of something on those two points, just sharing my experience using both in parallel.

I can't speak to how Veeam does it under the hood though. From using their PVE plugin, it seems like it's very much still a work in progress with a good amount of bug squashing going on.

 

[Johannes S]

We currently use both as we already had Veeam for ESXI, and it has some features that PBS does not support (yet?) Mostly in the Application Aware space.

I really like the PBS backup process and it's simplicity in getting set up. I especially like that there's not really a distinction between a full backup and a diff backup. I don't have to worry about making sure that my active fulls are aligned with the retention policy or making a synthetic full.

Where I find PBS lacking is in backing up things like SQL DBs as well as Veeam does. Veeam will interact with the DB itself, truncate logs, and you can do state restores of the DB without having to mess with the full VM.

But only if you don't need clustering for MySQL/MariaDB/Postgres. It never will fail to amaze me, that people are willing to accept this from a "enterprise backup" software. If however you need application aware stuff for MS SQL/AD a potential workaround is to treat the vms as if they were baremetal machines and install the Veeam agent in it. In fact I remember @Falk R. mentioned several times, that this combination (PBS for vm backup, a minimal subscription of Veeam_Agents for the Domain controllers/Windows Database servers) is his "to-go-setup" for price-sensitive customers since it's still cheaper than a full blown VEEAM-subscription.

Also, the Veeam file restore options are more robust. Being able to tell it to just put files back how they were is pretty awesome. on PBS, from what I've seen, you have to download the archive, and then restore it manually. With PVE, it does require the deployment of their agents for restore, which we didn't have to mess with on ESXI, but that's pretty minimal and can be done when needed.

It depends. You can restore single-files from the PBS backupstorage in the backup/restore screen but still need to reupload them afterwards:


Alternatively you could use the proxmox-back-client mount feature from the client (this works at least with Debian VMs). So yes still a little bit cumbersome, but you don't need to download the whole archive at least. This isn't possible from PBS itself if I recall correctly though.

I can't speak to how Veeam does it under the hood though. From using their PVE plugin, it seems like it's very much still a work in progress with a good amount of bug squashing going on.

I never tested it myself so take this with a grain of salts but the reports in this forum makes me assume that their ProxmoxVE support is more like a "public beta" than "enterprise-grade backup".

 

[bbgeek17]

Where I find PBS lacking is in backing up things like SQL DBs as well as Veeam does. Veeam will interact with the DB itself, truncate logs, and you can do state restores of the DB without having to mess with the full VM.

But only if you don't need clustering for MySQL/MariaDB/Postgres.

If however you need application aware stuff for MS SQL/AD a potential workaround is to treat the vms as if they were baremetal machines and install the Veeam agent in it

@Johannes S: I could be mistaken, but it appears to me that you and @iceph03nix are talking about the same thing : Veeam's agents.

@iceph03nix :
A dedicated backup/recovery company has years of head start here. It's in Veeam's DNA. Proper application handling is key in many environments, which is why Agent based backup will never go away.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Johannes S]

Proper application handling is key in many environments, which is why Agent based backup will never go away.

I actually agree with this. I wouldn't call the lack of support for SQL-Clusters "proper application handling" though.