Suggestion to speedup garbage collector

[freebee]

I'm using PBS for large amounts of data on HDD disks. During garbage collector processes, disks are extremely slow. SSDs are expensive in regions like mine. Perhaps something that speeds up the process is logging the backup chunks in an internal database (MySQL or SQLite).
The idea is that when deleting a backup, the database provides the path to the linked files to be removed, avoiding scans for the files and high iops. The database would only be used in cases of garbage collect, not being used for other things.

Best Regards.

 

[Dunuin]

GC needs to access millions of metadata entries so IOPS performance is needed. When using ZFS you can add a pair of SSDs as "special" vdevs to store your pools metadata. In that case you got the metadata on the fast SSDs and the data on the slow HDDs and the GC will only need a few minutes instead of a few hours. Something similar you could achieve by added a single SSD as a L2ARC with "secondarycache=metadata" ZFS option (but would only help with reads, not with async writes like a special vdev would do. So only GC would be faster but not backups/restores).

 

[dcsapak]

What @Dunuin said, also

The idea is that when deleting a backup, the database provides the path to the linked files to be removed, avoiding scans for the files and high iops. The database would only be used in cases of garbage collect, not being used for other things.

that would not work in any case, since the chunks are deduplicated across the whole datastore, so any index could use those chunks. to find out which chunks are not needed anymore we must check all backups

 

[itNGO]

Maybe this helps here also?
https://forum.proxmox.com/threads/p...arge-amount-of-backups-timeout-in-pve.111128/