[SOLVED] proxmox-backup-client to backup photos : Strange deduplication factor and long backup backup time after some days

[cryonie]

Hello,

Ok I understand what you say

Here is a new backup and ... everything is good now.

2021-03-24T10:18:11+01:00: starting new backup on datastore 'backup_raid10': "host/OMV/2021-03-24T09:18:11Z"
2021-03-24T10:18:11+01:00: download 'index.json.blob' from previous backup.
2021-03-24T10:18:11+01:00: created new dynamic index 1 ("host/OMV/2021-03-24T09:18:11Z/catalog.pcat1.didx")
2021-03-24T10:18:11+01:00: register chunks in 'photos.pxar.didx' from previous backup.
2021-03-24T10:18:11+01:00: download 'photos.pxar.didx' from previous backup.
2021-03-24T10:18:11+01:00: created new dynamic index 2 ("host/OMV/2021-03-24T09:18:11Z/photos.pxar.didx")
2021-03-24T11:12:18+01:00: Upload statistics for 'photos.pxar.didx'
2021-03-24T11:12:18+01:00: UUID: ea7c53d1572640d2bfa0181456c56940
2021-03-24T11:12:18+01:00: Checksum: 8fa78ea3e9a43335822be8afccb093311b360886e3b39511d5095cdabfd91e0d
2021-03-24T11:12:18+01:00: Size: 332049471290
2021-03-24T11:12:18+01:00: Chunk count: 87115
2021-03-24T11:12:18+01:00: Upload size: 0 (0%)
2021-03-24T11:12:18+01:00: Duplicates: 87115+0 (100%)
2021-03-24T11:12:18+01:00: successfully closed dynamic index 2
2021-03-24T11:12:18+01:00: register chunks in 'virginie.pxar.didx' from previous backup.
2021-03-24T11:12:18+01:00: download 'virginie.pxar.didx' from previous backup.
2021-03-24T11:12:18+01:00: created new dynamic index 3 ("host/OMV/2021-03-24T09:18:11Z/virginie.pxar.didx")
2021-03-24T11:18:08+01:00: Upload statistics for 'virginie.pxar.didx'
2021-03-24T11:18:08+01:00: UUID: f1b0b9f8013e4155b944503f4be97cd5
2021-03-24T11:18:08+01:00: Checksum: bac275dc32046d02c8f5e3169d7acdd5385d2639da88f15d56bd257739ad445c
2021-03-24T11:18:08+01:00: Size: 35343914756
2021-03-24T11:18:08+01:00: Chunk count: 9377
2021-03-24T11:18:08+01:00: Upload size: 0 (0%)
2021-03-24T11:18:08+01:00: Duplicates: 9377+0 (100%)
2021-03-24T11:18:08+01:00: successfully closed dynamic index 3
2021-03-24T11:18:08+01:00: Upload statistics for 'catalog.pcat1.didx'
2021-03-24T11:18:08+01:00: UUID: b6c3f78a67224d0f9c8517d95aa86271
2021-03-24T11:18:08+01:00: Checksum: 7e3cbd0fb2e43dd89f2663c09d9b6444b322486398bb0e5c6551be34d9a784de
2021-03-24T11:18:08+01:00: Size: 2979151
2021-03-24T11:18:08+01:00: Chunk count: 5
2021-03-24T11:18:08+01:00: Upload size: 2979151 (100%)
2021-03-24T11:18:08+01:00: Duplicates: 0+5 (100%)
2021-03-24T11:18:08+01:00: Compression: 30%
2021-03-24T11:18:08+01:00: successfully closed dynamic index 1
2021-03-24T11:18:08+01:00: add blob "/mnt/datastore/backup_raid10/host/OMV/2021-03-24T09:18:11Z/index.json.blob" (380 bytes, comp: 380)
2021-03-24T11:18:08+01:00: successfully finished backup
2021-03-24T11:18:08+01:00: backup finished successfully
2021-03-24T11:18:08+01:00: TASK OK

Right now my last question would be : how to know what backup uses what chunks ?
The idea is that i want to destroy backups that had the bug so i want to compare what chunks are used on what backup (from day to day) and i'll find the one that has many different chunks and be able to destroy it.
Is there a way for me to do it ?

Thanks

 

[fabian]

not directly.

given you know which client versions are affected (.9 and .10) and when you had those installed (see /var/log/apt/history.log), deleting snapshots from that time period should be sufficient (after making sure you don't mind not having backups covering that period, of course!).

you could also parse the .didx files (their format is rather simple, but binary: it starts with a 4k header, followed by entries consisting of an 8-byte little-endian (end-of-chunk) offset and a 32 byte chunk digest each) or wait for https://lists.proxmox.com/pipermail/pbs-devel/2021-February/002071.html to be finalized and available in a released version, and then compare/diff the chunk sets for each snapshot. in the end, you should end up with the same list covering the time period from above

 

[cryonie]

Ok thank you for all your information and your help
Marked this as solved.
Have a nice day