Proxmox Backup Server (PBS) 4.1.0 causing S3 InternalError (500) during Garbage Collection (GC)

[torroe]

I have a critical issue with Garbage Collection (GC) on my FileLu S3 datastore after updating Proxmox Backup Server to 4.1.0.

The GC worked perfectly before the kernel/PBS update.

️ Environment:​

• PBS Version: 4.1.0
• Kernel: 6.17.2-2-pve
• Datastore: S3 Endpoint (FileLu)

Problem:​

Backups and Verify jobs are successful. The GC job consistently fails during the sweep unused chunks phase with a server-side error:

XML

<span>&lt;<span>Error</span>&gt;</span><br><span>&lt;<span>Code</span>&gt;</span>InternalError<span>&lt;/<span>Code</span>&gt;</span><br><span>&lt;<span>Message</span>&gt;</span>Cannot write upload meta<span>&lt;/<span>Message</span>&gt;</span><br><span>&lt;/<span>Error</span>&gt;</span><br>TASK ERROR: unexpected status code 500 Internal Server Error<br>

Any advice is appreciated!

best regards
Torsten

 

[powersupport]

This problem started after upgrading Proxmox Backup Server to version 4.1.0. While backups and verification jobs continue to run successfully, the Garbage Collection (GC) task fails during the “sweep unused chunks” phase due to an error returned by the S3 storage provider:

InternalError: Cannot write upload meta (HTTP 500)

This behavior is related to a change in how PBS 4.1 performs garbage collection. The newer version relies on additional S3 metadata operations during GC. Unfortunately, the S3 endpoint used (FileLu) does not fully support these specific metadata operations, which causes the GC process to fail. This is a server-side limitation of the S3 provider, not a configuration or permission issue on your PBS server.

• Backup jobs and verification tasks are not affected and will continue to run normally.
• Only the Garbage Collection task is impacted.
• This issue did not occur in earlier PBS versions, as the GC process worked differently.

1. Recommended: Migrate the datastore to a fully S3-compatible storage provider (such as AWS S3, Wasabi, Backblaze B2, or a MinIO-based S3 service), which is known to work correctly with PBS 4.x.
2. Temporary workaround: Disable scheduled GC to prevent task failures (storage usage may increase over time).
3. Short-term option: Downgrade Proxmox Backup Server to a previous version where GC worked with this S3 endpoint (not recommended long-term).

 

[torroe]

Thank you very much for the fast and incredibly precise diagnosis! The confirmed information about the change in the GC logic in PBS 4.1.0 is extremely valuable.

I have already forwarded the API incompatibility diagnosis (specifically the issue with the new metadata operations) to FileLu support and requested a server-side adjustment from them.

I would genuinely like to continue using FileLu, as the performance and reliability have been excellent apart from this one GC issue.

Question to Proxmox Developers​

If FileLu is unable to support the necessary S3 metadata operations in the short term, or perhaps at all:

Would it be possible to introduce a provider quirk specifically for FileLu in a future PBS version?

The goal would be that when this S3 endpoint is detected, the Garbage Collection process could temporarily fall back to the older (working) method. This would allow users to utilize the newest PBS version while retaining full GC functionality, avoiding a forced migration to a different provider.

Thank you again for all the support!