Previous chunks don't seem to be used for new backups

[VictorSTS]

I've had some bad luck this time with a broken PBS server. This is the sequence of events:

- Cluster PVE does it's backups to PBS01 (v3.4.4) for nearly two years.
- PBS02 (v4.1.1) in a remote location syncs backups from PBS01. This has been working for like a year. It's on version 4.1.1 for at least a week now. There is like 25 unencrypted snapshots of each VM in PBS02.
- Yesterday, I enabled encrypted backups on PVE using new namespace of PBS01.
- Configured replica on PBS02 to sync the new encrypted namespace to a different namespace of PBS02 than that of the unencrypted backups.
- Literally a few hours later, PBS01 goes down with some hardware issue still being diagnosed.
- Backups for this cluster are critical, so I switched PVE to backup directly to PBS02 (1Gbps dark fiber, 3ms latency), without encryption in the hopes that It won't need to transfer much data as there are lots of already synced chunks from PBS01. Backups are done in a new namespace of the same datastore where the syncs are stored.

The problem I see is that during backup, every single chunk is transferred to PBS02 via the ptp fiber. It seems unable to reuse the existing chunks before sending them to PBS02. At the same time, the used space on PBS02 datastore is not increasing, meaning that even if they are sent to PBS02 it does not store them "again", because they are already in the datastore.

The backup log shows very low data reuse, around 10% in the best cases: i.e. "backup was done incrementally, reused 4.03 GiB (11%)".

I though (and had tested on lab setups) that during backup, PVE would read+compress+optionally encrypt every chunk, send it's hash to PBS and would only send the chunk to PBS if it isn't already there. What am I missing here?

 

[fabian]

please post a backup task log.

client-side deduplication can only happen if there is a previous snapshot on the backup target (datastore+namespace+group!) that is not a in a verification-failed state. based on your description I suspect you have created new backup groups in a new empty namespace, which means the first such backup (for each VM) will only do server-side deduplication (matching your description of transferring lots of data, but not taking up additional space).

 

[VictorSTS]

client-side deduplication can only happen if there is a previous snapshot on the backup target (datastore+namespace+group!)

This is what I was missing here. Yes, they are in a new, empty namespace. Now that I think it again makes sense as there is no "list of snapshots with their list of chunks" to compare to and help PBS decide before hand if a chunk should be transferred or not. Will do a local sync to the new namespace before proceeding with the next lot of backups.
Thanks!

 

[VictorSTS]

After a local sync from the namespace where PBS02 stored the synced snapshots to the namespace where PVE stores it's backups, now PVE only transfers changed data and in most cases even uses the existing dirty-map (I suspect dirty-map can be reused if a backup to the "new" encrypted namespace was not yet created). In this use case I also had to update the owner to the appropriate one for PVE to be able to read previous backups.