Stage 2 of GC finds no chunks (Hetzner S3)

M

Magg

New Member
Sep 23, 2026
6
0
1
Hello,

I have a remote PBS running in Hetzner. The backups originate from another PBS (pulls). The data is encrypted on the other PBS during backup.

Somehow it never deletes any chunks. I've deleted all snapshots of both a large VM, relative to my total size of about 1TB, and a relatively large CT, but this is not reflected in the storage usage. Object locking is not turned on in the S3 config AFAIK (but how do I check? The Hetzner Console is very limited). I used the standard settings when creating the bucket.

I saw the other Hetzner S3 thread, but I think this is a separate issue, as it mostly verifies well, but it just doesn't delete any chunks ever.

I have a rate limit in/out of 7M on the S3 endpoint and no bursting. This seems to work well.

Edit: Remove Vanished is checked, and I did delete the snapshots via the contents of the datastore on the remote server.

I have the impression that Stage 2 should cycle through all chunks?

Code: 
2026-09-23T17:20:54+03:00: marked 100% (441 of 441 index files)
2026-09-23T17:20:54+03:00: Start GC phase2 (sweep unused chunks)
2026-09-23T17:20:54+03:00: processed 0 total chunks
2026-09-23T17:20:54+03:00: Start GC phase3 (sweep unused chunk markers)
2026-09-23T17:20:54+03:00: processed 1% (3128 chunks)

Also the logs indicate that there are around 308.218 chunks, while the Hetzner console says 395.242 files (not sure if that's apples and oranges, or apples and apples?)

Code: 
2026-09-23T17:20:55+03:00: processed 99% (308218 chunks)
2026-09-23T17:20:55+03:00: Chunk cache: hits 33690542, misses 311299 (hit ratio 99.08%)
2026-09-23T17:20:55+03:00: Removed garbage: 0 B
2026-09-23T17:20:55+03:00: Removed chunks: 0
2026-09-23T17:20:55+03:00: Original data usage: 129.788 TiB
2026-09-23T17:20:55+03:00: On-Disk usage: 0 B (0.00%)
2026-09-23T17:20:55+03:00: On-Disk chunks: 0
2026-09-23T17:20:55+03:00: Deduplication factor: 1.00
2026-09-23T17:20:55+03:00: queued notification (id=uuid)
2026-09-23T17:20:55+03:00: TASK OK

Any ideas or advice?

Please find the GC logs attached, and config items below :)

Edit 2: The S3 Stats
S3 traffic:
Data uploaded 1.14 TiB
Data downloaded 4.80 TiB


S3 requests:

GET1677095
PUT395912
POST0
HEAD32
DELETE1

root@pbs-h:~# proxmox-backup-manager version --verbose
proxmox-backup 4.2.0 running kernel: 7.0.0-3-pve
proxmox-backup-server 4.2.6-1 running version: 4.2.0
proxmox-kernel-helper 9.2.0
proxmox-kernel-7.0 7.0.14-19
proxmox-kernel-7.0.0-3-pve-signed 7.0.0-3
ifupdown2 3.3.0-1+pmx12
libjs-extjs 7.0.0-7
proxmox-backup-docs 4.2.6-1
proxmox-backup-client 4.2.6-1
proxmox-mail-forward 1.0.3
proxmox-mini-journalreader 1.7
proxmox-offline-mirror-helper 0.7.4
proxmox-widget-toolkit 5.2.10
pve-xtermjs 6.0.0-2
smartmontools 7.5-pve2
zfsutils-linux 2.4.4-pve1

root@pbs-h:~# cat /etc/proxmox-backup/datastore.cfg
datastore: h-backup-vm-ct-01
backend bucket=h-backup-vm-ct-01,client=hetzner,type=s3
comment
gc-schedule daily
notification-mode notification-system
path /cache/h-backup-vm-ct-01
tuning gc-atime-safety-check=1

root@pbs-h:~# cat /etc/proxmox-backup/s3.cfg
s3-endpoint: hetzner
access-key <redacted>
endpoint hel1.your-objectstorage.com
rate-in 7 MiB
rate-out 7 MiB
region hel1
secret-key <redacted>
 

Attachments

Last edited:
Chris said:
Please run
Code: 
proxmox-backup-manager s3 check <s3-endpoint-id> <bucket>
and make sure to address your bucket correctly in the s3 endpoint config, see https://forum.proxmox.com/threads/hetzner-s3-object-store.171899/post-801588
Click to expand...

Thanks for getting back to me! I set the endpoint URL to what it is in the post you linked, and rebooted just for good measure. The GC job now took 25 seconds as opposed to 5 seconds the first time I ran it, then it took 5 seconds again. Please see attached logs.

I still does nothing in stage 2. Do I need to recreate the datastore?

I don't want to change the s3 endpoint url to the hardcoded value that the person in the thread you linked did, since I have another datastore also connected to the same s3 endpoint, but with a different bucket. Can do it if needed though?

This is the output of the command (nothing):
Code: 
root@pbs-h:~# proxmox-backup-manager s3 check hetzner h-backup-vm-ct-01
root@pbs-h:~#

I also tried with an invalid bucket just to see if there was any other output then:
Code: 
root@pbs-h:~# proxmox-backup-manager s3 check hetzner h-backup-vm-ct-01xxx
Error: head object failed

Caused by:
    bucket does not exist or no permission to access it
root@pbs-h:~#
 

Attachments

Hmm, the verify jobs are now failing

2026-09-24T12:23:43+03:00: verify h-backup-vm-ct-01:ct/102/2026-09-23T23:01:11Z
2026-09-24T12:23:43+03:00: check pct.conf.blob
2026-09-24T12:23:43+03:00: check root.pxar.didx
2026-09-24T12:23:56+03:00: chunk 32f9a06cb95fd4d74ad28a49d7ab1a2292d36fb4543d6a4d67676ee62ef40656 was marked as corrupt
2026-09-24T12:23:56+03:00: chunk 95a151fd08808ef8815d3683276ccd25bf9119ec340af9d9dd8252ba08b87817 was marked as corrupt
2026-09-24T12:23:56+03:00: chunk c68be1ed1bc7a947e315de4be5b2e1a76f0ba87436e0f0755de6fb56d966cda3 was marked as corrupt
2026-09-24T12:23:56+03:00: chunk 7aae1951bbd1ec7c5748d2c96c94b41365a2a587480dd638ef0a36513b969761 was marked as corrupt
2026-09-24T12:23:56+03:00: chunk e90a0345215043d5a3ca9c38b8d23f0c52c076a8e473ddf51f5d77e6cb8a60f1 was marked as corrupt
2026-09-24T12:23:56+03:00: chunk 3c7cac31d5b1f6d67df852568b8b3adf043e27edad5d62180c5fddafc10776c8 was marked as corrupt
2026-09-24T12:23:56+03:00: chunk 8b937983e0d49351ed6c1e095077987bd914bb5ba6b8dfcbc22fb3838aab6242 was marked as corrupt
2026-09-24T12:24:04+03:00: verified 0.00/0.00 MiB in 21.18 seconds, speed 0.00/0.00 MiB/s (275 errors)
2026-09-24T12:24:04+03:00: verify h-backup-vm-ct-01:ct/102/2026-09-23T23:01:11Z/root.pxar.didx failed: chunks could not be verified
2026-09-24T12:24:04+03:00: check catalog.pcat1.didx
2026-09-24T12:24:04+03:00: verified 0.00/0.00 MiB in 0.14 seconds, speed 0.00/0.00 MiB/s (2 errors)
2026-09-24T12:24:04+03:00: verify h-backup-vm-ct-01:ct/102/2026-09-23T23:01:11Z/catalog.pcat1.didx failed: chunks could not be verified
2026-09-24T12:24:05+03:00: Failed to verify the following snapshots/groups:
2026-09-24T12:24:05+03:00: ct/102/2026-09-23T23:01:11Z
2026-09-24T12:24:05+03:00: TASK ERROR: verification failed - please check the log for details
 
I would guess your chunks might not have been written to the intended location/bucket then? Check the respective datastore sub-path of the bucket. It should contain some chunk file objects with object keys of the form <datastore-name>/.chunks/<4hex-digest-prefix>/<chunk-digest>. Not sure where Hetzner routes the PUT requests to when the bucket name is not part of the request url. Do you have other buckets with the same access key?

Edit: Also please upgrade to the latest available version of PBS, you are still running 4.2.0, missing important security and bugfixes.
 
Last edited:
You are right, they are not in the <datastore-name>/ folder, but rather in the root of the bucket. I have two different buckets, with different access keys, and I see the other one was just populated inside the datastore folder instead of the bucket root after a small backup job.

I've updated! :)

Is there a way out of this with all functionality working for the chunks in the existing 1TB bucket, or will I have to delete and start over if I wanna get rid of the wrongly placed and garbage files?
 
You can try to move the .cnt and .chunks prefixed objects to the expected <datastore-name>/.cnt and <datastore-name>/.chunks prefix using provider side or third party tooling, from the PBS side you cannot adapt this, the datastore name is expected to be a prefix to all the object keys belonging to it.

Edit: Also, move .in-use to <datastore-name>/.in-use to protect your datastore from being incorrectly used by concurrent PBS instances.
 
Last edited:
Magg said:
You are right, they are not in the <datastore-name>/ folder, but rather in the root of the bucket. I have two different buckets, with different access keys, and I see the other one was just populated inside the datastore folder instead of the bucket root after a small backup job.
Click to expand...
By the way, this also explains why you didn't run into errors during initial setup and backup... Since you were using the same name for bucket and datastore, the datastore name got interpreted as the path-style bucket address https://<endpoint>/<bucket>. Will see if we can add additional check to the client to protect against this.
 
Cool! Thanks! I found the rclone move command, so I'm running that now. Looks promising.

Thanks for the assistance!
 
You must log in or register to reply here.
Top Bottom
Back