Cannot stop a "non running" garbage collection job

Dec 6, 2021
37
3
13
47
Hi,

We have the following odd behavior

1750432191870.png

root@backup:~# proxmox-backup-manager garbage-collection status rpool
┌──────────────────┬────────────────────────────────────────────────────────────────────────────────────┐
│ Name │ Value │
╞══════════════════╪════════════════════════════════════════════════════════════════════════════════════╡
│ store │ rpool │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ disk-bytes │ 18931220651253 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ disk-chunks │ 6574567 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ index-data-bytes │ 198197340232913 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ index-file-count │ 1162 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ pending-bytes │ 4558987282 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ pending-chunks │ 4071 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ removed-bad │ 0 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ removed-bytes │ 438480129688 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ removed-chunks │ 201328 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ still-bad │ 0 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ next-run │ 1750446000 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ schedule │ 21:00 │
├──────────────────┼────────────────────────────────────────────────────────────────────────────────────┤
│ upid │ UPID:backup:0000041B:00001559:00000031:684C7530:garbage_collection:rpool:root@pam: │
└──────────────────┴────────────────────────────────────────────────────────────────────────────────────┘


The logs show

ProxmoxBackup Server 3.4.1
()
2025-06-13T21:00:00+02:00: starting garbage collection on store rpool
2025-06-13T21:00:00+02:00: task triggered by schedule 'fri 21:00'
2025-06-13T21:00:01+02:00: Access time update check successful, proceeding with GC.
2025-06-13T21:00:01+02:00: Using access time cutoff 1d 5m, minimum access time is 2025-06-12T18:55:00Z
2025-06-13T21:00:01+02:00: Start GC phase1 (mark used chunks)
2025-06-13T21:00:49+02:00: marked 1% (12 of 1162 index files)
...
2025-06-14T04:39:33+02:00: marked 100% (1162 of 1162 index files)
2025-06-14T04:39:33+02:00: Start GC phase2 (sweep unused chunks)
2025-06-14T04:41:47+02:00: processed 1% (67435 chunks)
...
2025-06-14T08:03:37+02:00: processed 99% (6712061 chunks)
2025-06-14T08:05:43+02:00: Removed garbage: 408.366 GiB
2025-06-14T08:05:43+02:00: Removed chunks: 201328
2025-06-14T08:05:43+02:00: Pending removals: 4.246 GiB (in 4071 chunks)
2025-06-14T08:05:43+02:00: Original data usage: 180.259 TiB
2025-06-14T08:05:43+02:00: On-Disk usage: 17.218 TiB (9.55%)
2025-06-14T08:05:43+02:00: On-Disk chunks: 6574567
2025-06-14T08:05:43+02:00: Deduplication factor: 10.47
2025-06-14T08:05:43+02:00: Average chunk size: 2.746 MiB
2025-06-14T08:05:44+02:00: TASK OK

We are the 20th.

Trying to start another garbage collection fails with "already running".
ps doesn't help me find such a process to kill it.

We witness a high iodelay (20%). Healty zpool (raidz1 : 4x8Tb + 1Tb zcache)

PBS 3.4.1 amd64

Thanks for any help on this issue.
 
My bad, we can actually see that garbage collect job from the Dashboard page. It's at 29%.

No idea why it didn't show on the other page. Topic can be considered closed.