Backblaze S3: frequent errors

C

crazywolf13

Member
Oct 15, 2023
90
11
13
Hi,

I have set up sync jobs to backblaze s3, ran fine for a couple of weeks, but since I while I often get errors, it's not really clear if they are critical or not, how I can fix them or if they even matter.

Thanks for some information!

Errors look like this:

Code: 
2026-05-18T06:41:28+02:00: sync snapshot ct/162/2026-05-18T02:08:47Z
2026-05-18T06:41:28+02:00: sync archive pct.conf.blob
2026-05-18T06:41:28+02:00: sync archive fw.conf.blob
2026-05-18T06:41:29+02:00: sync archive root.pxar.didx
2026-05-18T06:49:20+02:00: removing backup snapshot "/mnt/backblaze-cache/ct/162/2026-05-18T02:08:47Z"
2026-05-18T06:49:20+02:00: percentage done: 80.28% (57/71 groups)
2026-05-18T06:49:20+02:00: sync group ct/162 failed - failed to upload chunk to s3 backend - upload failed: unexpected status code 500 Internal Server Error
2026-05-18T06:49:20+02:00: skipped: 4 snapshot(s) (2026-03-29T02:08:53Z .. 2026-05-10T02:09:48Z) - older than the newest snapshot present on sync target
2026-05-18T06:49:20+02:00: skipped: 4 snapshot(s) (2026-05-14T02:07:41Z .. 2026-05-17T02:07:14Z) - due to transfer-last
2026-05-18T06:49:20+02:00: sync snapshot ct/163/2026-05-18T02:07:19Z

or something like this:

Code: 
2026-05-18T05:59:07+02:00: sync snapshot ct/144/2026-05-18T02:07:12Z
2026-05-18T05:59:07+02:00: sync archive pct.conf.blob
2026-05-18T05:59:07+02:00: sync archive fw.conf.blob
2026-05-18T05:59:07+02:00: sync archive root.pxar.didx
2026-05-18T05:59:19+02:00: downloaded 3.284 MiB (272.455 KiB/s)
2026-05-18T05:59:19+02:00: sync archive catalog.pcat1.didx
2026-05-18T05:59:20+02:00: downloaded 265.25 KiB (2.096 MiB/s)
2026-05-18T05:59:20+02:00: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<Error>
    <Code>InternalError</Code>
    <Message>internal incident</Message>
</Error>

2026-05-18T05:59:20+02:00: sync snapshot ct/144/2026-05-18T02:07:12Z done
2026-05-18T05:59:20+02:00: percentage done: 54.93% (39/71 groups)
2026-05-18T05:59:20+02:00: skipped: 4 snapshot(s) (2026-03-29T02:11:04Z .. 2026-05-10T02:12:00Z) - older than the newest snapshot present on sync target
2026-05-18T05:59:20+02:00: skipped: 4 snapshot(s) (2026-05-14T02:11:42Z .. 2026-05-17T02:11:50Z) - due to transfer-last
2026-05-18T05:59:20+02:00: sync snapshot ct/145/2026-05-18T02:11:56Z

or

Code: 
2026-05-18T08:00:36+02:00: sync snapshot vm/126/2026-05-18T02:01:23Z
2026-05-18T08:00:36+02:00: sync archive qemu-server.conf.blob
2026-05-18T08:00:36+02:00: sync archive fw.conf.blob
2026-05-18T08:00:36+02:00: sync archive drive-tpmstate0-backup.img.fidx
2026-05-18T08:00:36+02:00: downloaded 8.013 KiB (31.305 KiB/s)
2026-05-18T08:00:37+02:00: sync archive drive-scsi0.img.fidx
2026-05-18T08:11:16+02:00: removing backup snapshot "/mnt/backblaze-cache/vm/126/2026-05-18T02:01:23Z"
2026-05-18T08:11:16+02:00: percentage done: 100.00% (71/71 groups)
2026-05-18T08:11:16+02:00: sync group vm/126 failed - failed to upload chunk to s3 backend - upload failed: unexpected status code 500 Internal Server Error
2026-05-18T08:11:16+02:00: Finished syncing root namespace, current progress: 70 groups, 1 snapshots
2026-05-18T08:11:16+02:00: queued notification (id=583a0237-ac1a-49a5-b148-0a1a9bb879a8)
2026-05-18T08:11:16+02:00: TASK ERROR: sync failed with some errors.


My config looks like this:

1779090871999.png
 
Hm, these look like errors that occur on Backblaze's end. Not sure if they expose this information to you, but if so can you see any errors reported in your Backblaze account? Or was there some kind of outage during the upload?

Looking at the code it seems that the s3 client has tried to upload the chunk up to three times [1], but failed each time. The last error reported here was a 500 internal server error, so that points at an issue on Backblaze's end.

[1]: https://git.proxmox.com/?p=proxmox....1b828c43fb951ae3f98562c8616db71f;hb=HEAD#l924
 
Hi, thanks for the quick reply!

Sadly I don't think any log or something like that is available on backblaze.

According to their statuspage there were no outages: https://status.backblaze.com/

I can also hardly believe that a service as big as backblaze can allow themself to have such frequent issues, as I think last 3 backups (last 3 weekly backups) all failed with this error.

Could this be some sort of network issue on my end, or is that a reply from backblaze?
Also could this be some sort of rate-limit?
 
crazywolf13 said:
Could this be some sort of network issue on my end, or is that a reply from backblaze?
Click to expand...
Do you use some kind of Proxy between your PBS host and Backblaze? If not, that seems unlikely.

crazywolf13 said:
Also could this be some sort of rate-limit?
Click to expand...
Looking at the docs for Backblaze [1] the error message here should be different. It should respond with 503 error, not a 500 one. So I doubt that this is the case.

However, in theory there could be an incompatibility between our s3 implementation and Backblaze's that isn't handled gracefully on their side. Since this error only happens sometimes, if I understand you correctly, it's difficult to tell what could be the exact culprit.

[1]: https://www.backblaze.com/docs/cloud-storage-rate-limits
 
Thanks for the replies,

Proxmox Backup Sever - 4.1.4

Indeed, the error seems to be the same, I have subscribed to the linked bug in bugzilla.
 
crazywolf13 said:
Thanks for the replies,

Proxmox Backup Sever - 4.1.4

Indeed, the error seems to be the same, I have subscribed to the linked bug in bugzilla.
Click to expand...
In that case I recommend to upgrade, the improvements to transient error handling were introduced with proxmox-backup-server 4.1.6-1
 
  • Like
Reactions: crazywolf13
You must log in or register to reply here.
Top Bottom
Back