Garbage Collect process hang a 67% in phase 1

[llinty]

Hello,

Garbage collect has not been running due to a nearly full FS on a NAS. Repository has been locked in read/only by the NAS itself. Now it is Read/Write again. I have pruned some jobs and started manually the Garbage Collect. It has been running for more than 1 day with 7TB of data.

Process is still up and running but does not seems to progress anymore.

2023-12-12T09:07:53+01:00: starting garbage collection on store NFSBackup
2023-12-12T09:08:13+01:00: Start GC phase1 (mark used chunks)
2023-12-12T09:13:03+01:00: marked 2% (1 of 46 index files)
2023-12-12T09:13:03+01:00: marked 4% (2 of 46 index files)
2023-12-12T09:34:17+01:00: marked 6% (3 of 46 index files)
2023-12-12T10:07:28+01:00: marked 8% (4 of 46 index files)
2023-12-12T10:07:28+01:00: marked 10% (5 of 46 index files)
2023-12-12T10:27:23+01:00: marked 13% (6 of 46 index files)
2023-12-12T10:58:43+01:00: marked 15% (7 of 46 index files)
2023-12-12T10:58:44+01:00: marked 17% (8 of 46 index files)
[...]
2023-12-13T05:22:07+01:00: marked 67% (31 of 46 index files)

Please advise because I'm no more able to free some space to make backup jobs up and running again!
Kind regards

 

[sb-jw]

Are you sure it hasn't been set back to read-only in the meantime?

 

[llinty]

Hi, no the storage is still read/write mode :-(

root@pbs:/var/log/proxmox-backup/tasks# dd if=/dev/zero bs=1M count=1 of=/opt/backup/1mega
1+0 enregistrements lus
1+0 enregistrements écrits
1048576 octets (1,0 MB, 1,0 MiB) copiés, 0,532175 s, 2,0 MB/s

root@pbs:/var/log/proxmox-backup/tasks# ls -altr /opt/backup/
total 2068
drwxr-xr-x 3 root root 4096 6 mars 2023 ..
-rw-r--r-- 1 backup backup 0 4 mai 2023 .lock
drwxr-x--- 1 backup backup 1052672 4 mai 2023 .chunks
drwxr-xr-x 9 backup backup 4096 15 juin 07:00 vm
-rw-r--r-- 1 backup backup 305 11 nov. 19:37 .gc-status
-rw-r--r-- 1 root root 1048576 13 déc. 16:18 1mega
drwxrwxrwx 5 backup backup 4096 13 déc. 16:18 .

 

[llinty]

and it is still running

root@pbs:/var/log/proxmox-backup/tasks# proxmox-backup-client task list --repository NFSBackup
Password for "root@pam": ***********
┌──────────────────────────┬─────────┬─────────────────────────────────────────────────────────────────────────────────────┬───────
│ starttime │ endtime │ upid │ status
╞══════════════════════════╪═════════╪═════════════════════════════════════════════════════════════════════════════════════╪═══════
│ Tue Dec 12 09:07:53 2023 │ │ UPIDbs:000002DD:000007A2:000000BC:657814D9:garbage_collection:NFSBackup:root@pam: │ runnin
└──────────────────────────┴─────────┴─────────────────────────────────────────────────────────────────────────────────────┴───────

 

[llinty]

Latest news the Garbage Collect finally arrived to Step 2 !!

It also started to remove unused chunks and the space used is now 60%. I don't know if the time spend is normal.

Do we have a way to do some performance tests with proxmox-backup-client on repository?

regards

 

[sb-jw]

It is also not recommended to use a data store on a NAS or generally a remote storage for PBS. It's best for the PBS to have local SSDs.

 

[llinty]

I agree for sure but this NAS is a secondary storage repository for backup not for running VMs.

 

[Dunuin]

GC performance will always be terrible unless metadata is stored or cached on SSDs.

 

[llinty]

GC performance will always be terrible unless metadata is stored or cached on SSDs.

It makes sense of course! How can I split metadata on SSD and chunks (what I call deduplicated data) on NAS?

 

[Dunuin]

This has to be done on the storage layer. Your NAS has to support some tiered storage/caching. If your NAS supports ZFS you could add some Enterprise SSDs to your NAS and use them as "special" vdevs or "cache" vdev with secondarycache=metadata.

 

[llinty]

This has to be done on the storage layer. You your NAS has to support some tiered storage/caching.If your NAS supports ZFS you could add some Enterprise SSDs to your NAS and use them as "special" vdevs or "cache" vdev with secondarycache=metadata.

Thank you for your explanation. Unfortunately customer is using an entry level NAS solution. However, I take a good note of your tips.