S3 Hetzner failures with Gargbage collection

[harunk]

Hello,

i have s3 datastore configured with Hetzner and the Garbage collection is failing consistently with the following error. The backups to s3 are running fine without any issues. Any ideas on what to check ?

Thanks!

2026-05-13T02:30:20+02:00: marked 99% (755 of 762 index files)
2026-05-13T02:30:20+02:00: marked 100% (762 of 762 index files)
2026-05-13T02:30:20+02:00: Start GC phase2 (sweep unused chunks)
2026-05-13T02:33:41+02:00: queued notification (id=ea91cebe-b5d6-4515-8ea5-81e880da4bce)
2026-05-13T02:33:41+02:00: TASK ERROR: error reading a body from connection: unexpected EOF during chunk size line

 

[Chris]

During phase 2 of garbage collection for datastores backed by S3, PBS iterates trough all the chunks present on the backend and deletes those which are no longer in use (detected during phase 1). The error would indicate that one of the response bodies for such a list object v2 S3 API is unexpectedly truncated.

How is the PBS host memory usage during garbage collection? Do you run into memory constrains? Is the PBS running on hardware or in a VM? If it is not memory constrained, most likely this is either a networking issue or an issue on the provider side.

 

[harunk]

The PBS is a standalone machine and according to the statistics page (see image below), there are no memory constraints. The s3 garbage collect runs at 2:30 everyday and here is the system summary from the Server administration page. Both cpu and memory have enough headroom available.

Interestingly, if it's a provider issue then the tasks that run before and after (backup, pruning) should also have some issues right? but they don't exhibit the same issue as here.

 

[Chris]

Interestingly, if it's a provider issue then the tasks that run before and after (backup, pruning) should also have some issues right?

Not necessarily: The GC perform a very specific API request with prefix and limit parameters during phase 2. Do you see any errors around the GC in the systemd journal? How is the network between you and the provider? Any proxies, VPN, ecc. which might interfere with the traffic?

Also, please post proxmox-backup-manager version --verbose

Edit: Also, please see if this applies to you: https://forum.proxmox.com/threads/f...s-code-504-gateway-timeout.176833/post-839329

 

[harunk]

Thanks for the response.


There are no proxies and the server is whitelisted in firewall . The logs in systemd show only the following

May 13 02:33:41 pbs0 proxmox-backup-proxy[137566]: TASK ERROR: error reading a body from connection: unexpected EOF during chunk size line
May 13 02:33:44 pbs0 proxmox-backup-api[137538]: notified via target `mail-to-root`

proxmox-backup-manager version --verbose

proxmox-backup-manager version --verbose
proxmox-backup                      4.2.0         running kernel: 6.17.13-2-pve
proxmox-backup-server               4.2.0-1       running version: 4.2.0       
proxmox-kernel-helper               9.0.4                                     
proxmox-kernel-7.0                  7.0.2-2                                   
proxmox-kernel-7.0.2-2-pve-signed   7.0.2-2                                   
proxmox-kernel-7.0.0-3-pve-signed   7.0.0-3                                   
proxmox-kernel-6.17.13-7-pve-signed 6.17.13-7                                 
proxmox-kernel-6.17                 6.17.13-7                                 
proxmox-kernel-6.17.13-6-pve-signed 6.17.13-6                                 
proxmox-kernel-6.17.13-4-pve-signed 6.17.13-4                                 
proxmox-kernel-6.17.13-2-pve-signed 6.17.13-2                                 
proxmox-kernel-6.17.13-1-pve-signed 6.17.13-1                                 
proxmox-kernel-6.17.2-1-pve-signed  6.17.2-1                                   
ifupdown2                           3.3.0-1+pmx12                             
libjs-extjs                         7.0.0-5                                   
proxmox-backup-docs                 4.2.0-1                                   
proxmox-backup-client               4.2.0-1                                   
proxmox-mail-forward                1.0.3                                     
proxmox-mini-journalreader          1.6                                       
proxmox-offline-mirror-helper       0.7.3                                     
proxmox-widget-toolkit              5.1.9                                     
pve-xtermjs                         5.5.0-3                                   
smartmontools                       7.4-pve1                                   
zfsutils-linux                      2.4.1-pve1

Edit: Also, please see if this applies to you: https://forum.proxmox.com/threads/f...s-code-504-gateway-timeout.176833/post-839329

This bucket is new and was created i think only a couple of months back so don't think we are affected by this issue.

 

[Chris]

Internal testing with a S3 bucket hosted at Hetzner showed no issues. Could you also share the network config? I would also suggest to reach out to Hetzner, they might have more insights if the connection is prematurely closed on there side.