[SOLVED] Looking for a way to do incremental backups without having to read all the data every time.

[itachi737]

Hi,

I'm trying to figure out a way to do incremental backup of my media library (8tb) without having to read the whole 8tb every single time. Right now, the 8tb are in a zpool on my pve. From there I attached them to a lxc that is running the proxmox backup client via a script in order to back it up to the pbs which is installed on a on another pc (bare-metal) that also uses zfs. The initial backup worked fine. It did the whole 8tb in 2 days. My problem is that the incremental backups also take a considerable amount of time. I'm running my first one right now (I only added one new file 5gb) and it's been running for more than 2h. (I'll update the exact time it takes once it finishes). (It took 9h)

This is the command I currently use: proxmox-backup-client backup backups.pxar:/mnt/backups media.pxar:/mnt/media files.pxar:/mnt/files

When I first installed it, I was under the impression that the file level backup looks at the files that have been modified. To do so I thought it was reading the last modified part, but after doing some research on the forum, it seems that it reads the whole file.

Is there any way that I can avoid having read all the data? Since this is media data it will mostly never change.

If there is no way to do it via pbs, is there another way to do it that you recommend?

Thank you.

 

[B.Otto]

From what I gather you use the PBS-client on a machine to back up directories directly? Then I dont think it is possible to use incremental backups.

Iirc incremental backups are only supported when backing up full Virtual Machines running on a PVE server, since then the QEMU service managing the VM can keep track of the changed blocks in a 'dirty bitmap' table.

I don't know how you use your data, but it would work if you put it into a VM running on your PVE server and then backup the whole VM on your PBS server.

 

[fabian]

incremental backups work (the read data is detected as duplicate and not re-uploaded), but the extra read overhead is unavoidable at the moment for host/ct backups.

 

[itachi737]

Yes, the duplicate data is indeed not being sent over, just the new data, but it still has to read everything so it can take quiet a bit of time. Thank you for the clarification.

 

[t.oster]

Hi, would it be possible (protocol wise) to write a proxmox-backup-client specifically for zfs filesystems which can:

on the first backup:
- make a snapshot before a backup
- backup the data to PBS from that snapshot
=> backup is "atomic"

on subsequent backups:
- make a new snapshot
- ask zfs (through zed or anything) which blocks have changed
- only process (read) files/blocks which have changed and do report all other blocks as unchanged
=> No matter how much TB your FS is, if you only touch one file, only that file has to be read from disk

So the general idea is to make the same improvements which are in VMs through the dirty-bitmap-table with some ZFS magic.

IMHO the big use case would be network shares/media libraries etc where you have tons of data which practically never change. ZFS already keeps a list of changed blocks and it would be great if this could be used.

I do not know the internals of the PBS protocol, nor everything about ZFS but my general question is:
- Does anything in the PBS protocol make this impossible
- Can ZFS be efficiently queried about a list of changed blocks/files without reading the whole dataset?

 

[fabian]

Hi, would it be possible (protocol wise) to write a proxmox-backup-client specifically for zfs filesystems which can:

on the first backup:
- make a snapshot before a backup
- backup the data to PBS from that snapshot
=> backup is "atomic"

on subsequent backups:
- make a new snapshot
- ask zfs (through zed or anything) which blocks have changed
- only process (read) files/blocks which have changed and do report all other blocks as unchanged
=> No matter how much TB your FS is, if you only touch one file, only that file has to be read from disk

technically yes (it is similar to the approach that @Chris is currently fleshing out), but see below

So the general idea is to make the same improvements which are in VMs through the dirty-bitmap-table with some ZFS magic.

the problem is that this is nowhere near as efficient as dirty bitmap tracking - ZFS doesn't magically know which files have changed, it needs to check all the records to find out which files have changed. this is not very efficient (it is more efficient than reading all the data to then decide it is still the same, but it is less efficient than the approach we are currently implementing which is filesystem-agnostic).

IMHO the big use case would be network shares/media libraries etc where you have tons of data which practically never change. ZFS already keeps a list of changed blocks and it would be great if this could be used.

I do not know the internals of the PBS protocol, nor everything about ZFS but my general question is:
- Does anything in the PBS protocol make this impossible

it requires quite extensive adaption on the archiver/client side, which Chris has been working on for quite a while and which is starting to shape up

- Can ZFS be efficiently queried about a list of changed blocks/files without reading the whole dataset?

not really. our current approach (splitting metadata from data, and then using the previous metadata to quickly scan the input dir for re-usable entries) is faster than zfs diff for many inputs, and has the benefit of not being ZFS specific nor requiring coordination of storage snapshots and backup runs.

you can find the last series on our devel list, but please be aware that this is not yet fully tested or reviewed, and everything including on-disk file formats are still subject to change! DO NOT USE THIS ANYWHERE NEAR PRODUCTION DATA, IT MAY EAT IT.

https://lists.proxmox.com/pipermail/pbs-devel/2024-March/008236.html

 

[t.oster]

Hi, thank you for the quick answer. Looks interesting, I hope it will be production ready soon, so I can start testing.

Right now we are using a zfs send/receive approach for that kind of data, which is really performant, but I understand that this works on ZFS record-level which may not be translated efficiently to file/blocklevel.

As far as I understand, your mentioned approach still needs a fast connection to the PBS server, so it may still be beneficial for some specific workloads to be able to determine all changed blocks on the client without any server-communication (zfs diff), imagine a 10TB dataset where only a few MB have been added/changed. Can you estimate how such workload would perform with your mentioned approach? How much data would be read from the disk, how much data has to be fetched from the server and be sent to it?

 

[fabian]

the current approach:
- the previous snapshots index file needs to be downloaded and read
- all files which are backed up need to be stat-ed and fully read and chunked
- for each chunk it's then possible to look up in the previous index whether the server already has it, if it does -> no need to upload

the new approach
- the previous snapshot's index files needs to be downloaded and read
- the previous snapshot's metadata archive needs to be downloaded
- all files which are backed up are stat-ed
-- if the metadata is unchanged compared to the previous run, the file (or rather, its contents) is selected for re-use
-- file contents where the metadata hasn't changed which are stored in (mostly) re-used chunks will re-use those chunks (no reading of data -> this is where the speedup comes from!)
-- file contents where the metadata has changed are read and encoded into new chunks (but some or all of their previous contents might still be part of another file's reused chunk as "padding")
-- file contents where the metadata hasn't changed, but re-using the chunk would result in too much unneeded data (padding) of other files which have changed are also read and re-encoded into new chunks

the last is an optimization to avoid space usage blowing up over time.

the ratio of metadata to data of course depends on your input (one of my test inputs is ~5Mio smallish files that make up ~55GB of data - the corresponding metadata archive that needs to be fetched is 830MB uncompressed, but it compresses fairly well and is only about 100MB on disk/on the wire). an incremental run with the patch set and a few hundred added/changed/moved/.. files takes about 1 minute, whereas a regular run takes 10-12 minutes (both on fast storage). and this is a very metadata skewed workload, for bigger files the penalty for the metadata lookups (both on-disk, and from the previous snapshot) is of course smaller and the speedup should be bigger.

 

[Dunuin]

Thats for the detailed summary. Always nice to get a better understanding of how things work under the hood without reading endless pages of source code. And of cause to know what is worked on right know and why.