Deduplication question

[tsumaru720]

Hello

I've set up a couple of new LXC containers recently, all using the same base OS and they have all been initialised with similar packages.

I noticed however that during the first backup of my most recent container - the output seemed to suggest that it had to back up all of it, and didnt reuse any existing chunks - How does the dedupe work in this case?

I'd have expected it to have found some matching chunks from the existing containers of the same OS/version with similar packages? I know the dedupe works well for further snapshots of the same container, but I was confused as to the initial backup state reported through PVE

 

[Chris]

HI,
the PBS client gets the index of the previous snapshot and does not re-upload chunks already present in that index file. This is the incremental nature of the PBS client.
However, if chunks are not present in the previous index file or if there is no previous index, these chunks will have to be uploaded to the server.

On the server side however, these chunks can be de-duplicated in the datastore, as hash and therefore content is the same.
Please check the deduplication factor of your datastore in the PBS WebUI. This tells you how well data chunks could be re-referenced.

 

[tsumaru720]

oh ok, so even though the client reported that it had to back up the entire container, thats just the upload to the PBS server and it was likely deduplicated as it was written? I just dont get visibility of how much dedupe happened because there's no previous index for this CT i guess?

 

[Chris]

oh ok, so even though the client reported that it had to back up the entire container, thats just the upload to the PBS server and it was likely deduplicated as it was written? I just dont get visibility of how much dedupe happened because there's no previous index for this CT i guess?

Yes, the client must not know about the chunks in the datastore and deduplication happening there.