Increased backup size of VMs

[z3t4]

Hi,
I've recently deployed a pbs node, it's awesome.

I noticed that the reported VM backup size on PVE GUI is the size of the vm disks, not the size of the used space on those disks like on the old backups, so the backup sizes of all my VMs multiplied threefold.

Backups on containers also doubled size, but are well below their disk size, more like their used space but way higher.

Is there any way to compress, or reduce the backup size ignoring the unused space? Maybe it is not reporting the real disk space used to store the backups, and the de-duplication reduces it;

How can I check how many disk space really uses a backup?, It is possible to export it on the old format to restore a host outside of pbs reach?

Thanks

 

[fabian]

the backup task log will show you how much data it generated before and after deduplication and compression.

 

[z3t4]

Yes, it does, thanks.

So PBS considers whole disk images for de-duplication.

But I can't see how many space does use all the backups of a vm, I suppose that the first one would be big, and the subsequent ones just deltas.

Maybe also doesn't make much sense, as deleting a backup may or may not free disk space, as the chunks it is using may be also be used on other backups.

So deleting a backup that can restore a 100GB disk could just free some MB, up to the 100GB, depending on the uniqueness of the chunks it uses.

In that case, why PBS reports the container backup sizes lower than the disk images? maybe those backups are done at a filesystem level like the ones done with the client instead of block level?

Thanks & kind regards.

 

[fabian]

Yes, it does, thanks.

So PBS considers whole disk images for de-duplication.

for VMs: the disk images are chunked (into 4MB chunks), those are then deduplicated at three levels - 1st, if the VM is running and has a valid bitmap with changed chunks since the last backup, only those chunks are created and considered. 2nd, all chunks already referenced by the previous backup snapshot are generated, but not uploaded to the server. 3rd, the server discards chunks it already has instead of writing them to disk again.

But I can't see how many space does use all the backups of a vm, I suppose that the first one would be big, and the subsequent ones just deltas.

the first one is big as in it will upload more data, but after that all backups are equal since a backup snapshot just consists of a list of chunks + some metadata.

Maybe also doesn't make much sense, as deleting a backup may or may not free disk space, as the chunks it is using may be also be used on other backups.

exactly. this is also why it would be prohibitively expensive to always have a current view of each snapshots usage, since each new or removed snapshot would require recalculating all snapshots' usage (or keeping lots of additional data around to update instead of recalculate, neither is efficient for bigger datastores).

So deleting a backup that can restore a 100GB disk could just free some MB, up to the 100GB, depending on the uniqueness of the chunks it uses.

exactly

In that case, why PBS reports the container backup sizes lower than the disk images? maybe those backups are done at a filesystem level like the ones done with the client instead of block level?

also true

host and ct backups use the same mechanism - first, a filesystem/directory is translated into a 'pxar' archive (a custom archive format, this step requires reading ALL the data), this archive is then chunked on-the-fly (with an algorithm trying to split the chunks at meaningful boundaries so that subsequent backups end up with similar boundaries and hopefully lots of identical chunks), the client again skips uploading chunks which are already present in the previous snapshot, and the server discards all chunks it already has stored.

 

[z3t4]

Thank you for the explanation!