Garbage Collect take long

helpfinder · Oct 2, 2024

Hi folks,
I have read some threads about GC here in forum, but was not able to get any answer to my question/issue

for me it seems Garbage Collect takes quite long.
I have PC where I am running Proxmox VE and on this PVE I have 3 VMs:
1. Docker (Ubuntu)
2. Home assistant
3. PBS

On PBS I have scheduled daily backups of first 2 mentioned VMs and storage is NFS share on my Synology NAS

Garbage Collect log (job takes more than 2 hours)

Code:

2024-10-02T00:00:00+02:00: starting garbage collection on store Backup_proxmox
2024-10-02T00:00:00+02:00: task triggered by schedule 'daily'
2024-10-02T00:00:00+02:00: Start GC phase1 (mark used chunks)
2024-10-02T00:04:47+02:00: marked 1% (2 of 162 index files)
2024-10-02T00:07:43+02:00: marked 2% (4 of 162 index files)
2024-10-02T00:09:53+02:00: marked 3% (5 of 162 index files)
2024-10-02T00:11:10+02:00: marked 4% (7 of 162 index files)
2024-10-02T00:12:33+02:00: marked 5% (9 of 162 index files)
2024-10-02T00:13:14+02:00: marked 6% (10 of 162 index files)
2024-10-02T00:13:56+02:00: marked 7% (12 of 162 index files)
2024-10-02T00:13:57+02:00: marked 8% (13 of 162 index files)
2024-10-02T00:14:39+02:00: marked 9% (15 of 162 index files)
2024-10-02T00:14:39+02:00: marked 10% (17 of 162 index files)
2024-10-02T00:15:39+02:00: marked 11% (18 of 162 index files)
2024-10-02T00:16:16+02:00: marked 12% (20 of 162 index files)
2024-10-02T00:18:03+02:00: marked 13% (22 of 162 index files)
2024-10-02T00:19:40+02:00: marked 14% (23 of 162 index files)
2024-10-02T00:22:41+02:00: marked 15% (25 of 162 index files)
2024-10-02T00:24:12+02:00: marked 16% (26 of 162 index files)
2024-10-02T00:27:04+02:00: marked 17% (28 of 162 index files)
2024-10-02T00:30:04+02:00: marked 18% (30 of 162 index files)
2024-10-02T00:31:45+02:00: marked 19% (31 of 162 index files)
2024-10-02T00:36:18+02:00: marked 20% (33 of 162 index files)
2024-10-02T00:40:05+02:00: marked 21% (35 of 162 index files)
2024-10-02T00:41:38+02:00: marked 22% (36 of 162 index files)
2024-10-02T00:46:44+02:00: marked 23% (38 of 162 index files)
2024-10-02T00:48:46+02:00: marked 24% (39 of 162 index files)
2024-10-02T00:51:26+02:00: marked 25% (41 of 162 index files)
2024-10-02T00:54:00+02:00: marked 26% (43 of 162 index files)
2024-10-02T00:55:11+02:00: marked 27% (44 of 162 index files)
2024-10-02T00:58:04+02:00: marked 28% (46 of 162 index files)
2024-10-02T00:59:08+02:00: marked 29% (47 of 162 index files)
2024-10-02T01:02:24+02:00: marked 30% (49 of 162 index files)
2024-10-02T01:05:44+02:00: marked 31% (51 of 162 index files)
2024-10-02T01:07:08+02:00: marked 32% (52 of 162 index files)
2024-10-02T01:10:17+02:00: marked 33% (54 of 162 index files)
2024-10-02T01:13:01+02:00: marked 34% (56 of 162 index files)
2024-10-02T01:14:10+02:00: marked 35% (57 of 162 index files)
2024-10-02T01:16:30+02:00: marked 36% (59 of 162 index files)
2024-10-02T01:17:38+02:00: marked 37% (60 of 162 index files)
2024-10-02T01:20:01+02:00: marked 38% (62 of 162 index files)
2024-10-02T01:22:27+02:00: marked 39% (64 of 162 index files)
2024-10-02T01:23:38+02:00: marked 40% (65 of 162 index files)
2024-10-02T01:26:52+02:00: marked 41% (67 of 162 index files)
2024-10-02T01:29:37+02:00: marked 42% (69 of 162 index files)
2024-10-02T01:29:37+02:00: marked 43% (70 of 162 index files)
2024-10-02T01:30:18+02:00: marked 44% (72 of 162 index files)
2024-10-02T01:30:47+02:00: marked 45% (73 of 162 index files)
2024-10-02T01:31:58+02:00: marked 46% (75 of 162 index files)
2024-10-02T01:32:35+02:00: marked 47% (77 of 162 index files)
2024-10-02T01:32:35+02:00: marked 48% (78 of 162 index files)
2024-10-02T01:33:11+02:00: marked 49% (80 of 162 index files)
2024-10-02T01:33:45+02:00: marked 50% (81 of 162 index files)
2024-10-02T01:34:38+02:00: marked 51% (83 of 162 index files)
2024-10-02T01:35:28+02:00: marked 52% (85 of 162 index files)
2024-10-02T01:35:28+02:00: marked 53% (86 of 162 index files)
2024-10-02T01:36:10+02:00: marked 54% (88 of 162 index files)
2024-10-02T01:36:50+02:00: marked 55% (90 of 162 index files)
2024-10-02T01:37:28+02:00: marked 56% (91 of 162 index files)
2024-10-02T01:38:07+02:00: marked 57% (93 of 162 index files)
2024-10-02T01:38:07+02:00: marked 58% (94 of 162 index files)
2024-10-02T01:38:47+02:00: marked 59% (96 of 162 index files)
2024-10-02T01:39:25+02:00: marked 60% (98 of 162 index files)
2024-10-02T01:40:14+02:00: marked 61% (99 of 162 index files)
2024-10-02T01:40:42+02:00: marked 62% (101 of 162 index files)
2024-10-02T01:41:20+02:00: marked 63% (103 of 162 index files)
2024-10-02T01:41:20+02:00: marked 64% (104 of 162 index files)
2024-10-02T01:41:54+02:00: marked 65% (106 of 162 index files)
2024-10-02T01:42:36+02:00: marked 66% (107 of 162 index files)
2024-10-02T01:43:32+02:00: marked 67% (109 of 162 index files)
2024-10-02T01:44:09+02:00: marked 68% (111 of 162 index files)
2024-10-02T01:44:09+02:00: marked 69% (112 of 162 index files)
2024-10-02T01:44:55+02:00: marked 70% (114 of 162 index files)
2024-10-02T01:45:53+02:00: marked 71% (116 of 162 index files)
2024-10-02T01:46:32+02:00: marked 72% (117 of 162 index files)
2024-10-02T01:47:05+02:00: marked 73% (119 of 162 index files)
2024-10-02T01:47:05+02:00: marked 74% (120 of 162 index files)
2024-10-02T01:47:38+02:00: marked 75% (122 of 162 index files)
2024-10-02T01:48:18+02:00: marked 76% (124 of 162 index files)
2024-10-02T01:48:56+02:00: marked 77% (125 of 162 index files)
2024-10-02T01:49:34+02:00: marked 78% (127 of 162 index files)
2024-10-02T01:49:34+02:00: marked 79% (128 of 162 index files)
2024-10-02T01:50:19+02:00: marked 80% (130 of 162 index files)
2024-10-02T01:50:59+02:00: marked 81% (132 of 162 index files)
2024-10-02T01:51:32+02:00: marked 82% (133 of 162 index files)
2024-10-02T01:52:12+02:00: marked 83% (135 of 162 index files)
2024-10-02T01:52:44+02:00: marked 84% (137 of 162 index files)
2024-10-02T01:52:44+02:00: marked 85% (138 of 162 index files)
2024-10-02T01:53:24+02:00: marked 86% (140 of 162 index files)
2024-10-02T01:54:02+02:00: marked 87% (141 of 162 index files)
2024-10-02T01:54:39+02:00: marked 88% (143 of 162 index files)
2024-10-02T01:55:13+02:00: marked 89% (145 of 162 index files)
2024-10-02T01:55:13+02:00: marked 90% (146 of 162 index files)
2024-10-02T01:55:50+02:00: marked 91% (148 of 162 index files)
2024-10-02T01:56:42+02:00: marked 92% (150 of 162 index files)
2024-10-02T01:57:11+02:00: marked 93% (151 of 162 index files)
2024-10-02T01:57:52+02:00: marked 94% (153 of 162 index files)
2024-10-02T01:57:52+02:00: marked 95% (154 of 162 index files)
2024-10-02T01:58:33+02:00: marked 96% (156 of 162 index files)
2024-10-02T01:59:09+02:00: marked 97% (158 of 162 index files)
2024-10-02T01:59:42+02:00: marked 98% (159 of 162 index files)
2024-10-02T02:01:20+02:00: marked 99% (161 of 162 index files)
2024-10-02T02:01:21+02:00: marked 100% (162 of 162 index files)
2024-10-02T02:01:21+02:00: found (and marked) 141 index files outside of expected directory scheme
2024-10-02T02:01:21+02:00: Start GC phase2 (sweep unused chunks)
2024-10-02T02:02:01+02:00: processed 1% (1077 chunks)
2024-10-02T02:02:05+02:00: processed 2% (2140 chunks)
2024-10-02T02:02:09+02:00: processed 3% (3233 chunks)
2024-10-02T02:02:13+02:00: processed 4% (4319 chunks)
2024-10-02T02:02:18+02:00: processed 5% (5407 chunks)
2024-10-02T02:02:22+02:00: processed 6% (6500 chunks)
2024-10-02T02:02:26+02:00: processed 7% (7555 chunks)
2024-10-02T02:02:30+02:00: processed 8% (8678 chunks)
2024-10-02T02:02:35+02:00: processed 9% (9742 chunks)
2024-10-02T02:02:39+02:00: processed 10% (10792 chunks)
2024-10-02T02:02:43+02:00: processed 11% (11943 chunks)
2024-10-02T02:02:47+02:00: processed 12% (13045 chunks)
2024-10-02T02:02:51+02:00: processed 13% (14117 chunks)
2024-10-02T02:02:56+02:00: processed 14% (15151 chunks)
2024-10-02T02:03:00+02:00: processed 15% (16246 chunks)
2024-10-02T02:03:04+02:00: processed 16% (17330 chunks)
2024-10-02T02:03:08+02:00: processed 17% (18476 chunks)
2024-10-02T02:03:13+02:00: processed 18% (19639 chunks)
2024-10-02T02:03:17+02:00: processed 19% (20748 chunks)
2024-10-02T02:03:21+02:00: processed 20% (21800 chunks)
2024-10-02T02:03:25+02:00: processed 21% (22864 chunks)
2024-10-02T02:03:30+02:00: processed 22% (23952 chunks)
2024-10-02T02:03:34+02:00: processed 23% (25046 chunks)
2024-10-02T02:03:38+02:00: processed 24% (26125 chunks)
2024-10-02T02:03:42+02:00: processed 25% (27256 chunks)
2024-10-02T02:03:46+02:00: processed 26% (28316 chunks)
2024-10-02T02:03:50+02:00: processed 27% (29434 chunks)
2024-10-02T02:03:55+02:00: processed 28% (30599 chunks)
2024-10-02T02:03:59+02:00: processed 29% (31673 chunks)
2024-10-02T02:04:03+02:00: processed 30% (32676 chunks)
2024-10-02T02:04:07+02:00: processed 31% (33794 chunks)
2024-10-02T02:04:11+02:00: processed 32% (34876 chunks)
2024-10-02T02:04:16+02:00: processed 33% (35909 chunks)
2024-10-02T02:04:20+02:00: processed 34% (37015 chunks)
2024-10-02T02:04:24+02:00: processed 35% (38061 chunks)
2024-10-02T02:04:28+02:00: processed 36% (39177 chunks)
2024-10-02T02:04:33+02:00: processed 37% (40317 chunks)
2024-10-02T02:04:37+02:00: processed 38% (41393 chunks)
2024-10-02T02:04:41+02:00: processed 39% (42484 chunks)
2024-10-02T02:04:45+02:00: processed 40% (43527 chunks)
2024-10-02T02:04:49+02:00: processed 41% (44532 chunks)
2024-10-02T02:04:53+02:00: processed 42% (45560 chunks)
2024-10-02T02:04:57+02:00: processed 43% (46600 chunks)
2024-10-02T02:05:01+02:00: processed 44% (47699 chunks)
2024-10-02T02:05:06+02:00: processed 45% (48710 chunks)
2024-10-02T02:05:10+02:00: processed 46% (49729 chunks)
2024-10-02T02:05:14+02:00: processed 47% (50786 chunks)
2024-10-02T02:05:19+02:00: processed 48% (51867 chunks)
2024-10-02T02:05:23+02:00: processed 49% (52959 chunks)
2024-10-02T02:05:27+02:00: processed 50% (54047 chunks)
2024-10-02T02:05:32+02:00: processed 51% (55124 chunks)
2024-10-02T02:05:36+02:00: processed 52% (56164 chunks)
2024-10-02T02:05:40+02:00: processed 53% (57240 chunks)
2024-10-02T02:05:44+02:00: processed 54% (58354 chunks)
2024-10-02T02:05:49+02:00: processed 55% (59412 chunks)
2024-10-02T02:05:53+02:00: processed 56% (60551 chunks)
2024-10-02T02:05:57+02:00: processed 57% (61637 chunks)
2024-10-02T02:06:01+02:00: processed 58% (62691 chunks)
2024-10-02T02:06:05+02:00: processed 59% (63777 chunks)
2024-10-02T02:06:10+02:00: processed 60% (64806 chunks)
2024-10-02T02:06:14+02:00: processed 61% (65901 chunks)
2024-10-02T02:06:18+02:00: processed 62% (66933 chunks)
2024-10-02T02:06:22+02:00: processed 63% (67984 chunks)
2024-10-02T02:06:26+02:00: processed 64% (69048 chunks)
2024-10-02T02:06:31+02:00: processed 65% (70175 chunks)
2024-10-02T02:06:35+02:00: processed 66% (71235 chunks)
2024-10-02T02:06:39+02:00: processed 67% (72339 chunks)
2024-10-02T02:06:44+02:00: processed 68% (73444 chunks)
2024-10-02T02:06:48+02:00: processed 69% (74485 chunks)
2024-10-02T02:06:53+02:00: processed 70% (75536 chunks)
2024-10-02T02:06:57+02:00: processed 71% (76663 chunks)
2024-10-02T02:07:01+02:00: processed 72% (77741 chunks)
2024-10-02T02:07:06+02:00: processed 73% (78849 chunks)
2024-10-02T02:07:10+02:00: processed 74% (79956 chunks)
2024-10-02T02:07:15+02:00: processed 75% (81059 chunks)
2024-10-02T02:07:19+02:00: processed 76% (82120 chunks)
2024-10-02T02:07:24+02:00: processed 77% (83213 chunks)
2024-10-02T02:07:28+02:00: processed 78% (84344 chunks)
2024-10-02T02:07:33+02:00: processed 79% (85478 chunks)
2024-10-02T02:07:37+02:00: processed 80% (86589 chunks)
2024-10-02T02:07:41+02:00: processed 81% (87698 chunks)
2024-10-02T02:07:46+02:00: processed 82% (88705 chunks)
2024-10-02T02:07:50+02:00: processed 83% (89815 chunks)
2024-10-02T02:07:55+02:00: processed 84% (90861 chunks)
2024-10-02T02:07:59+02:00: processed 85% (91938 chunks)
2024-10-02T02:08:03+02:00: processed 86% (93059 chunks)
2024-10-02T02:08:08+02:00: processed 87% (94049 chunks)
2024-10-02T02:08:12+02:00: processed 88% (95152 chunks)
2024-10-02T02:08:16+02:00: processed 89% (96249 chunks)
2024-10-02T02:08:21+02:00: processed 90% (97368 chunks)
2024-10-02T02:08:25+02:00: processed 91% (98452 chunks)
2024-10-02T02:08:29+02:00: processed 92% (99513 chunks)
2024-10-02T02:08:34+02:00: processed 93% (100582 chunks)
2024-10-02T02:08:38+02:00: processed 94% (101683 chunks)
2024-10-02T02:08:42+02:00: processed 95% (102783 chunks)
2024-10-02T02:08:47+02:00: processed 96% (103902 chunks)
2024-10-02T02:08:51+02:00: processed 97% (104960 chunks)
2024-10-02T02:08:56+02:00: processed 98% (106041 chunks)
2024-10-02T02:09:00+02:00: processed 99% (107125 chunks)
2024-10-02T02:09:05+02:00: Removed garbage: 0 B
2024-10-02T02:09:05+02:00: Removed chunks: 0
2024-10-02T02:09:05+02:00: Pending removals: 814.246 MiB (in 709 chunks)
2024-10-02T02:09:05+02:00: Original data usage: 4.089 TiB
2024-10-02T02:09:05+02:00: On-Disk usage: 128.546 GiB (3.07%)
2024-10-02T02:09:05+02:00: On-Disk chunks: 107489
2024-10-02T02:09:05+02:00: Deduplication factor: 32.57
2024-10-02T02:09:05+02:00: Average chunk size: 1.225 MiB
2024-10-02T02:09:05+02:00: TASK OK

Backup takes few minutes - 2-3 minutes

Is ths normal? Or how long the GC prune should take?

Thanks

waltar · Oct 2, 2024

GC needs fast metadata performance so best is to have the backup "disks" directly connected to PBS as a bare metal installation, or as pci passthrough or if used behind nfs the filesystem on the nfs server should use any kind of ssd's or if on rust disks should use a metadata special device as with zfs or xfs is possible.

helpfinder · Oct 2, 2024

thanks for reply
that said, it seems it is normal - just my setup is not fast enough
Now wondering if I really need to do this 2+ hours excercise every night or I can reachedule prune job once per week or so

tcabernoch · Oct 5, 2024

You can run your Prune and GC Jobs as infrequently as you deem necessary.
How full is your datastore? Do you have room for more data? PBS does some stellar dedupe, you can probably run once a week. Just watch how much space it uses and tune your pruning interval accordingly.
Just so you know, this is happening because of the NFS mount. Local storage takes exponentially less time. 5-10 minutes on a big system.

helpfinder · Oct 7, 2024

Datastore is full about 74% currently - I am not struggling with low disk space

Search

Search

Garbage Collect take long

helpfinder

New Member

waltar

Active Member

helpfinder

New Member

tcabernoch

Active Member

helpfinder

New Member