GC very slow after 3.4 update

[rauth]

Hi,

As reported in Proxmox Backup Server 3.4 release forum, I am facing extreme slowness in GC task after upgrade: Here

proxmox-backup-manager version --verbose

proxmox-backup                     3.4.0         running kernel: 6.8.12-9-pve
proxmox-backup-server              3.4.1-1       running version: 3.4.1
proxmox-kernel-helper              8.1.1
pve-kernel-6.2                     8.0.5
proxmox-kernel-6.8                 6.8.12-10
proxmox-kernel-6.8.12-9-pve-signed 6.8.12-9
proxmox-kernel-6.8.4-3-pve-signed  6.8.4-3
proxmox-kernel-6.2.16-20-pve       6.2.16-20
proxmox-kernel-6.2                 6.2.16-20
pve-kernel-6.2.16-3-pve            6.2.16-3
ifupdown2                          3.2.0-1+pmx11
libjs-extjs                        7.0.0-5
proxmox-backup-docs                3.4.1-1
proxmox-backup-client              3.4.1-1
proxmox-mail-forward               0.3.2
proxmox-mini-journalreader         1.4.0
proxmox-offline-mirror-helper      unknown
proxmox-widget-toolkit             4.3.10
pve-xtermjs                        5.5.0-2
smartmontools                      7.3-pve1
zfsutils-linux                     2.2.7-pve2

cat /etc/proxmox-backup/prune.cfg

datastore: pbs01
        gc-schedule sat 05:00
        notification-mode notification-system
        path /mnt/datastore/pbs01
        tuning gc-atime-safety-check=1

I don't have any pruning tasks configured on the PBS side, but rather it is configured in each job on the proxmox side, I have different retentions for each VM. There are currently 384 Groups and 2138 Snapshots, approximately 1.8TiB of backups are inserted every day

 

[Chris]

Hi,
thanks for sharing the outputs.
First off, from the output you showed in the other thread one can see that there are missing chunks in your datastore

2025-05-12T13:39:01-03:00: warning: unable to access non-existent chunk 780f82546e91812e1b1c04815c2afbf73ad963bc99650502b8c2b379093e3407, required by "/mnt/datastore/pbs002/ns/cloudkvm02/vm/20510/2025-05-12T00:55:08Z/drive-scsi0.img.fidx"<br>2025-05-12T13:39:01-03:00: warning: unable to access non-existent chunk 6a8c3eea53c7a6f96418f0b48f7c47ac342497836d3c3c9b6fe6a5d7826ed353, required by "/mnt/datastore/pbs002/ns/cloudkvm02/vm/20510/2025-05-12T00:55:08Z/drive-scsi0.img.fidx"

So it is strongly recommended to run a verification job, which however may take some time in your case.

Regarding the other log lines:

2025-05-13T01:29:04-03:00: Encountered new index file '"/mnt/datastore/pbs/ns/stor01/vm/20288/2025-05-12T06:00:02Z/drive-scsi0.img.fidx"', increment total index file count<br>2025-05-13T01:37:01-03:00: Encountered new index file '"/mnt/datastore/pbs/ns/stor01/vm/20266/2025-05-12T02:26:25Z/drive-scsi0.img.fidx"', increment total index file count<br>2025-05-13T01:37:46-03:00: Encountered new index file '"/mnt/datastore/pbs/ns/stor01/vm/20266/2025-05-11T02:11:07Z/drive-scsi0.img.fidx"', increment total index file count<br>

These indicate that new backup snapshots where created while the GC was ongoing. Can you please share the retention settings for VMID 20288 and VMID 20266 as well as the VM config, qm config <VMID>?

Also, did you already try to increase the gc-cache-capacity in the datastore tuning options to the maximum value of 8388608 as suggested? Does this have a significant runtime? See https://pbs.proxmox.com/docs/storage.html#tuning