Pending Data is not being removed by the GC tasks

[supportlooker]

Hello,

I noticed recently that our backup disk space is starting to run low.
The GC tasks does seem to be running but there are several GB of Pending data that are not being removed.
What is the best way to handle this? I had read that directly trying to clear the Pending Data manually could lead to
back up loss.

Is there a best practice for how to handle this?

 

[cappa_]

Good question. I am facing the same in our PBS instance. Since a month the usage of one of the disk is increasing and now reached nearly 100% before a month the estimated full was "never" since about a year. I checked the size of the VM's but they are pretty the same like months ago. The GC says 0B removed on the last job and I would like to remove the "Pending Data" which is a lot but safe it is to do it manually?

 

[Chris]

Hi,
please share the full task log for the garbage collection as well as your proxmox-backup-manager version --verbose. Did you prune any backup snapshots before running the garbage collection, did the approx. 24h pass in-between 2 garbage collection runs?

Please note that the atime of the chunk files is used to identify unused chunks for garbage collection. See https://pbs.proxmox.com/docs/backup-client.html#garbage-collection

 

[cappa_]

Hi. Just share my GC logs:

2024-09-04T07:34:39+00:00: starting garbage collection on store nvme1-512
2024-09-04T07:34:39+00:00: Start GC phase1 (mark used chunks)
2024-09-04T07:34:39+00:00: marked 2% (1 of 50 index files)
2024-09-04T07:34:39+00:00: marked 4% (2 of 50 index files)
2024-09-04T07:34:39+00:00: marked 6% (3 of 50 index files)
2024-09-04T07:34:39+00:00: marked 8% (4 of 50 index files)
2024-09-04T07:34:39+00:00: marked 10% (5 of 50 index files)
2024-09-04T07:34:39+00:00: marked 12% (6 of 50 index files)
2024-09-04T07:34:39+00:00: marked 14% (7 of 50 index files)
2024-09-04T07:34:39+00:00: marked 16% (8 of 50 index files)
2024-09-04T07:34:39+00:00: marked 18% (9 of 50 index files)
2024-09-04T07:34:39+00:00: marked 20% (10 of 50 index files)
2024-09-04T07:34:40+00:00: marked 22% (11 of 50 index files)
2024-09-04T07:34:40+00:00: marked 24% (12 of 50 index files)
2024-09-04T07:34:40+00:00: marked 26% (13 of 50 index files)
2024-09-04T07:34:40+00:00: marked 28% (14 of 50 index files)
2024-09-04T07:34:40+00:00: marked 30% (15 of 50 index files)
2024-09-04T07:34:40+00:00: marked 32% (16 of 50 index files)
2024-09-04T07:34:40+00:00: marked 34% (17 of 50 index files)
2024-09-04T07:34:40+00:00: marked 36% (18 of 50 index files)
2024-09-04T07:34:40+00:00: marked 38% (19 of 50 index files)
2024-09-04T07:34:40+00:00: marked 40% (20 of 50 index files)
2024-09-04T07:34:40+00:00: marked 42% (21 of 50 index files)
2024-09-04T07:34:40+00:00: marked 44% (22 of 50 index files)
2024-09-04T07:34:40+00:00: marked 46% (23 of 50 index files)
2024-09-04T07:34:41+00:00: marked 48% (24 of 50 index files)
2024-09-04T07:34:41+00:00: marked 50% (25 of 50 index files)
2024-09-04T07:34:41+00:00: marked 52% (26 of 50 index files)
2024-09-04T07:34:41+00:00: marked 54% (27 of 50 index files)
2024-09-04T07:34:41+00:00: marked 56% (28 of 50 index files)
2024-09-04T07:34:41+00:00: marked 58% (29 of 50 index files)
2024-09-04T07:34:41+00:00: marked 60% (30 of 50 index files)
2024-09-04T07:34:41+00:00: marked 62% (31 of 50 index files)
2024-09-04T07:34:41+00:00: marked 64% (32 of 50 index files)
2024-09-04T07:34:41+00:00: marked 66% (33 of 50 index files)
2024-09-04T07:34:41+00:00: marked 68% (34 of 50 index files)
2024-09-04T07:34:41+00:00: marked 70% (35 of 50 index files)
2024-09-04T07:34:41+00:00: marked 72% (36 of 50 index files)
2024-09-04T07:34:41+00:00: marked 74% (37 of 50 index files)
2024-09-04T07:34:41+00:00: marked 76% (38 of 50 index files)
2024-09-04T07:34:42+00:00: marked 78% (39 of 50 index files)
2024-09-04T07:34:42+00:00: marked 80% (40 of 50 index files)
2024-09-04T07:34:42+00:00: marked 82% (41 of 50 index files)
2024-09-04T07:34:42+00:00: marked 84% (42 of 50 index files)
2024-09-04T07:34:42+00:00: marked 86% (43 of 50 index files)
2024-09-04T07:34:42+00:00: marked 88% (44 of 50 index files)
2024-09-04T07:34:42+00:00: marked 90% (45 of 50 index files)
2024-09-04T07:34:42+00:00: marked 92% (46 of 50 index files)
2024-09-04T07:34:42+00:00: marked 94% (47 of 50 index files)
2024-09-04T07:34:42+00:00: marked 96% (48 of 50 index files)
2024-09-04T07:34:43+00:00: marked 98% (49 of 50 index files)
2024-09-04T07:34:43+00:00: marked 100% (50 of 50 index files)
2024-09-04T07:34:43+00:00: Start GC phase2 (sweep unused chunks)
2024-09-04T07:34:43+00:00: processed 1% (1709 chunks)
2024-09-04T07:34:43+00:00: processed 2% (3410 chunks)
2024-09-04T07:34:43+00:00: processed 3% (5181 chunks)
2024-09-04T07:34:43+00:00: processed 4% (6857 chunks)
2024-09-04T07:34:43+00:00: processed 5% (8627 chunks)
2024-09-04T07:34:43+00:00: processed 6% (10409 chunks)
2024-09-04T07:34:43+00:00: processed 7% (12211 chunks)
2024-09-04T07:34:43+00:00: processed 8% (13941 chunks)
2024-09-04T07:34:43+00:00: processed 9% (15756 chunks)
2024-09-04T07:34:43+00:00: processed 10% (17523 chunks)
2024-09-04T07:34:43+00:00: processed 11% (19270 chunks)
2024-09-04T07:34:43+00:00: processed 12% (21021 chunks)
2024-09-04T07:34:43+00:00: processed 13% (22710 chunks)
2024-09-04T07:34:43+00:00: processed 14% (24453 chunks)
2024-09-04T07:34:43+00:00: processed 15% (26252 chunks)
2024-09-04T07:34:43+00:00: processed 16% (28125 chunks)
2024-09-04T07:34:43+00:00: processed 17% (30389 chunks)
2024-09-04T07:34:43+00:00: processed 18% (32931 chunks)
2024-09-04T07:34:43+00:00: processed 19% (35846 chunks)
2024-09-04T07:34:43+00:00: processed 20% (39060 chunks)
2024-09-04T07:34:43+00:00: processed 21% (42389 chunks)
2024-09-04T07:34:43+00:00: processed 22% (45679 chunks)
2024-09-04T07:34:43+00:00: processed 23% (48988 chunks)
2024-09-04T07:34:43+00:00: processed 24% (52370 chunks)
2024-09-04T07:34:43+00:00: processed 25% (55690 chunks)
2024-09-04T07:34:43+00:00: processed 26% (58867 chunks)
2024-09-04T07:34:43+00:00: processed 27% (62086 chunks)
2024-09-04T07:34:43+00:00: processed 28% (65495 chunks)
2024-09-04T07:34:43+00:00: processed 29% (68829 chunks)
2024-09-04T07:34:43+00:00: processed 30% (72166 chunks)
2024-09-04T07:34:43+00:00: processed 31% (75439 chunks)
2024-09-04T07:34:43+00:00: processed 32% (78877 chunks)
2024-09-04T07:34:43+00:00: processed 33% (82210 chunks)
2024-09-04T07:34:43+00:00: processed 34% (85504 chunks)
2024-09-04T07:34:43+00:00: processed 35% (88798 chunks)
2024-09-04T07:34:43+00:00: processed 36% (92046 chunks)
2024-09-04T07:34:43+00:00: processed 37% (95400 chunks)
2024-09-04T07:34:43+00:00: processed 38% (98756 chunks)
2024-09-04T07:34:43+00:00: processed 39% (102085 chunks)
2024-09-04T07:34:43+00:00: processed 40% (105342 chunks)
2024-09-04T07:34:43+00:00: processed 41% (108637 chunks)
2024-09-04T07:34:43+00:00: processed 42% (112005 chunks)
2024-09-04T07:34:43+00:00: processed 43% (115284 chunks)
2024-09-04T07:34:43+00:00: processed 44% (118596 chunks)
2024-09-04T07:34:43+00:00: processed 45% (121886 chunks)
2024-09-04T07:34:43+00:00: processed 46% (125258 chunks)
2024-09-04T07:34:43+00:00: processed 47% (128585 chunks)
2024-09-04T07:34:43+00:00: processed 48% (131823 chunks)
2024-09-04T07:34:43+00:00: processed 49% (135111 chunks)
2024-09-04T07:34:43+00:00: processed 50% (138412 chunks)
2024-09-04T07:34:43+00:00: processed 51% (141704 chunks)
2024-09-04T07:34:43+00:00: processed 52% (145007 chunks)
2024-09-04T07:34:43+00:00: processed 53% (148366 chunks)
2024-09-04T07:34:43+00:00: processed 54% (151601 chunks)
2024-09-04T07:34:43+00:00: processed 55% (154858 chunks)
2024-09-04T07:34:43+00:00: processed 56% (158161 chunks)
2024-09-04T07:34:43+00:00: processed 57% (161437 chunks)
2024-09-04T07:34:43+00:00: processed 58% (164764 chunks)
2024-09-04T07:34:43+00:00: processed 59% (168027 chunks)
2024-09-04T07:34:43+00:00: processed 60% (171397 chunks)
2024-09-04T07:34:43+00:00: processed 61% (174678 chunks)
2024-09-04T07:34:43+00:00: processed 62% (178002 chunks)
2024-09-04T07:34:43+00:00: processed 63% (181322 chunks)
2024-09-04T07:34:43+00:00: processed 64% (184604 chunks)
2024-09-04T07:34:43+00:00: processed 65% (187906 chunks)
2024-09-04T07:34:43+00:00: processed 66% (191330 chunks)
2024-09-04T07:34:43+00:00: processed 67% (194642 chunks)
2024-09-04T07:34:43+00:00: processed 68% (197807 chunks)
2024-09-04T07:34:43+00:00: processed 69% (201079 chunks)
2024-09-04T07:34:43+00:00: processed 70% (204392 chunks)
2024-09-04T07:34:43+00:00: processed 71% (207746 chunks)
2024-09-04T07:34:43+00:00: processed 72% (210998 chunks)
2024-09-04T07:34:44+00:00: processed 73% (214227 chunks)
2024-09-04T07:34:44+00:00: processed 74% (217572 chunks)
2024-09-04T07:34:44+00:00: processed 75% (220843 chunks)
2024-09-04T07:34:44+00:00: processed 76% (224119 chunks)
2024-09-04T07:34:44+00:00: processed 77% (227397 chunks)
2024-09-04T07:34:44+00:00: processed 78% (230705 chunks)
2024-09-04T07:34:44+00:00: processed 79% (233985 chunks)
2024-09-04T07:34:44+00:00: processed 80% (237342 chunks)
2024-09-04T07:34:44+00:00: processed 81% (240731 chunks)
2024-09-04T07:34:44+00:00: processed 82% (244142 chunks)
2024-09-04T07:34:44+00:00: processed 83% (247439 chunks)
2024-09-04T07:34:44+00:00: processed 84% (250680 chunks)
2024-09-04T07:34:44+00:00: processed 85% (254056 chunks)
2024-09-04T07:34:44+00:00: processed 86% (257373 chunks)
2024-09-04T07:34:44+00:00: processed 87% (260630 chunks)
2024-09-04T07:34:44+00:00: processed 88% (263838 chunks)
2024-09-04T07:34:44+00:00: processed 89% (267107 chunks)
2024-09-04T07:34:44+00:00: processed 90% (270400 chunks)
2024-09-04T07:34:44+00:00: processed 91% (273698 chunks)
2024-09-04T07:34:44+00:00: processed 92% (276960 chunks)
2024-09-04T07:34:44+00:00: processed 93% (280269 chunks)
2024-09-04T07:34:44+00:00: processed 94% (283655 chunks)
2024-09-04T07:34:44+00:00: processed 95% (286980 chunks)
2024-09-04T07:34:44+00:00: processed 96% (290377 chunks)
2024-09-04T07:34:44+00:00: processed 97% (293658 chunks)
2024-09-04T07:34:44+00:00: processed 98% (297021 chunks)
2024-09-04T07:34:44+00:00: processed 99% (300320 chunks)
2024-09-04T07:34:44+00:00: Removed garbage: 0 B
2024-09-04T07:34:44+00:00: Removed chunks: 0
2024-09-04T07:34:44+00:00: Pending removals: 278.469 GiB (in 209798 chunks)
2024-09-04T07:34:44+00:00: Original data usage: 4.414 TiB
2024-09-04T07:34:44+00:00: On-Disk usage: 152.497 GiB (3.37%)
2024-09-04T07:34:44+00:00: On-Disk chunks: 93869
2024-09-04T07:34:44+00:00: Deduplication factor: 29.64
2024-09-04T07:34:44+00:00: Average chunk size: 1.664 MiB
2024-09-04T07:34:44+00:00: TASK OK

And the verbose output of `proxmox-backup-manager version`:

> proxmox-backup-manager version --verbose
...
proxmox-backup                     3.2.0        running kernel: 6.8.8-2-pve
proxmox-backup-server              3.2.3-1      running version: 3.2.3
proxmox-kernel-helper              8.1.0
pve-kernel-6.2                     8.0.5
proxmox-kernel-6.8                 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed  6.8.8-2
proxmox-kernel-6.8.4-3-pve-signed  6.8.4-3
proxmox-kernel-6.8.4-2-pve-signed  6.8.4-2
proxmox-kernel-6.5.13-5-pve-signed 6.5.13-5
proxmox-kernel-6.5                 6.5.13-5
proxmox-kernel-6.5.11-7-pve-signed 6.5.11-7
proxmox-kernel-6.5.11-6-pve-signed 6.5.11-6
proxmox-kernel-6.2.16-20-pve       6.2.16-20
proxmox-kernel-6.2                 6.2.16-20
pve-kernel-6.2.16-3-pve            6.2.16-3
ifupdown2                          3.2.0-1+pmx8
libjs-extjs                        7.0.0-4
proxmox-backup-docs                3.2.3-1
proxmox-backup-client              3.2.3-1
proxmox-mail-forward               0.2.3
proxmox-mini-journalreader         1.4.0
proxmox-offline-mirror-helper      unknown
proxmox-widget-toolkit             4.2.3
pve-xtermjs                        5.3.0-3
smartmontools                      7.3-pve1
zfsutils-linux                     2.2.4-pve1

There is a prune job scheduled hourly and the GC is scheduled to run once a week on Saturday.

The disks are mounted with this options in `/etc/mtab`:

/dev/nvme1n1p1 /mnt/datastore/nvme1-512 ext4 rw,relatime 0 0

Is it a better way to mount the drives with the `atime` option?

 

[Chris]

The disks are mounted with this options in `/etc/mtab`:

/dev/nvme1n1p1 /mnt/datastore/nvme1-512 ext4 rw,relatime 0 0

Is it a better way to mount the drives with the `atime` option?

No, it is fine to use relatime as mount option. This is why the 24h + 5min grace period is used.

Is something touching the chunks in your chunk store in-between the 2 garbage collection runs? Maybe do you have an external process which accesses the chunks?

 

[cappa_]

Okay, thank you very much. Yes we are using the s3cmd to sync the drive and it looks like that a s3cmd sync job is touching the data. Thanks again pointing it out.

 

[Chris]

Okay that does explain why the garbage collection is not able to clean up the chunks then.

How often do you sync to S3? Maybe you can trigger an additional garbage collection so that chunks can be cleaned up, so a pattern like S3 sync > GC (updating atime) > GC (actually cleans up chunks after 24h + 5min) > S3 sync?
Or increase the frequency of the GC if the additional IO load is not of concern?