Issues with BackBlaze and PBS4

Kenticles

New Member
Aug 14, 2025
4
0
1
Hello everyone. I am trying to get my Backblaze B2 and PBS4 to work. I am able to setup an endpoint and point to the correct bucket. Whenever I try to back everything up, it gives a chunk error. My understanding is that the local cache downloads everything first and then uploads to S3. With this in mind, I increased the cache to 256GB. I am able to successfully add the datastore through my PVE GUI. I am running PBS4 in a VM on my PVE cluster. Here is the log of a failed job, says the same thing no matter what VM/LXC I try to back up. Any help is appreciated!

EDIT: I am still running PVE 8.4.11 if that matters.


()
INFO: starting new backup job: vzdump 107 --node KServer1 --notification-mode auto --notes-template '{{guestname}}' --mode snapshot --remove 0 --storage PBS
INFO: Starting Backup of VM 107 (lxc)
INFO: Backup started at 2025-08-14 00:28:04
INFO: status = running
INFO: CT Name: Anchor-1
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
WARNING: You have not turned on protection against thin pools running out of space.
WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
Logical volume "snap_vm-107-disk-0_vzdump" created.
WARNING: Sum of all thin volume sizes (<120.50 GiB) exceeds the size of thin pool pve/data and the size of whole volume group (<118.24 GiB).
INFO: creating Proxmox Backup Server archive 'ct/107/2025-08-14T04:28:04Z'
INFO: set max number of entries in memory for file-based backups to 1048576
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp3225230_107/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 107 --backup-time 1755145684 --entries-max 1048576 --repository root@pam@pbs:mystorage
INFO: Starting backup: ct/107/2025-08-14T04:28:04Z
INFO: Client name: KServer1
INFO: Starting backup protocol: Thu Aug 14 00:28:04 2025
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp3225230_107/etc/vzdump/pct.conf' to 'root@pam@pbs:8007:mystorage' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@pbs:8007:mystorage' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: unclosed encoder dropped
INFO: closed encoder dropped with state
INFO: unfinished encoder state dropped
INFO: Error: failed to upload chunk to s3 backend
INFO: cleanup temporary 'vzdump' snapshot
Logical volume "snap_vm-107-disk-0_vzdump" successfully removed.
ERROR: Backup of VM 107 failed - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp3225230_107/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 107 --backup-time 1755145684 --entries-max 1048576 --repository root@pam@pbs:mystorage' failed: exit code 255
INFO: Failed at 2025-08-14 00:28:26
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors
 
Hi,
INFO: Error: failed to upload chunk to s3 backend
please check and post an excerpt of the systemd journal from around the time of the failed chunk upload. Do you maybe run into request/storage limits of the free tier?

Whenever I try to back everything up, it gives a chunk error. My understanding is that the local cache downloads everything first and then uploads to S3. With this in mind, I increased the cache to 256GB. I am able to successfully add the datastore through my PVE GUI. I am running PBS4 in a VM on my PVE cluster. Here is the log of a failed job, says the same thing no matter what VM/LXC I try to back up. Any help is appreciated!
No, the local cache does not need to hold all the chunks, otherwise this would not make much sense. The available cache slots to store chunks locally are calculated at datastore instantiation, older chunks being evicted once the capacity is reached and new chunks need to be cached.
 
Hi,

please check and post an excerpt of the systemd journal from around the time of the failed chunk upload. Do you maybe run into request/storage limits of the free tier?


No, the local cache does not need to hold all the chunks, otherwise this would not make much sense. The available cache slots to store chunks locally are calculated at datastore instantiation, older chunks being evicted once the capacity is reached and new chunks need to be cachedHey

Hi, this is the log I got when running the job:

2025-08-18T09:12:51-04:00: starting new backup on datastore 'mystorage' from ::ffff:192.168.1.25: "ct/107/2025-08-18T16:06:51Z"
2025-08-18T09:12:51-04:00: protocol upgrade done
2025-08-18T09:12:51-04:00: GET /previous_backup_time
2025-08-18T09:12:51-04:00: POST /blob
2025-08-18T09:12:51-04:00: Uploaded blob to object store: .cnt/ct/107/2025-08-18T16:06:51Z/pct.conf.blob
2025-08-18T09:12:51-04:00: add blob "/opt/cache/ct/107/2025-08-18T16:06:51Z/pct.conf.blob" (242 bytes, comp: 242)
2025-08-18T09:12:51-04:00: POST /dynamic_index
2025-08-18T09:12:51-04:00: POST /dynamic_index
2025-08-18T09:12:51-04:00: created new dynamic index 2 ("ct/107/2025-08-18T16:06:51Z/root.pxar.didx")
2025-08-18T09:12:51-04:00: created new dynamic index 1 ("ct/107/2025-08-18T16:06:51Z/catalog.pcat1.didx")
2025-08-18T09:12:51-04:00: POST /dynamic_chunk
2025-08-18T09:12:51-04:00: Upload of new chunk c484af0cca204e8f07bcd410e63d3d731d1f8ce12767fb81044a1057c696591c
2025-08-18T09:12:51-04:00: POST /dynamic_chunk
2025-08-18T09:12:51-04:00: Upload of new chunk 6e3c4049d5483f7e73b47e8cc8c7f99e56df57793778bd71f8a7a101e768b54c
2025-08-18T09:12:51-04:00: POST /dynamic_chunk
2025-08-18T09:12:51-04:00: Upload of new chunk e52e21a7cf443849e14f534edc23d26fe3c8d165408a7bbeb2a2c1adfdf81265
2025-08-18T09:12:51-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 37517f5044d52805460c017b0f810aa92ea048fe54612079260f21ca2f55c038
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk d62367067a78933d31901b5245ec321510bdacff2039e518f3fb5a2d3255b9ab
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk ef8e9ba2a3064ea06b81bd92c01172af0d4160a67977fd0208f1ef21241de2f5
2025-08-18T09:12:52-04:00: Upload of new chunk 8344d80cba4f6d880ce3e5641f52ef7ecf502ac3b6baf432d8a08e15a2cebc75
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 0c0882e58791f738b53046581777200993d42bd5e5b0bc8a1bc8e85d82259fcd
2025-08-18T09:12:52-04:00: Upload of new chunk 9163b95fd7289fc2754c2846ac4446f13b70e51537d29ee23004f71b791ab18e
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 3dad5ebe3dbb78aeef93596290ee7c921ecf2bc4597567ea23020d9c2bd59682
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk a192f87d08beec1cc6927fbae5a41dec3a166131b409c2edb2819c40ad4f0698
2025-08-18T09:12:52-04:00: Upload of new chunk a7f4ac1dc034f0f2034e9c6f666c7801e09d3f6ef2013b2ef2ee3f86b6dff0e4
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 31ff6f4653718c03b1217d003f8dfc021ab1b8093275cfacccf5891c17270ca1
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk b639ad31511d97ee983f7f94b128a1a9811f7dd94eaae261ef54189b3505937e
2025-08-18T09:12:52-04:00: Upload of new chunk ddaa590ff37b89693d9933c3f8d0246d97e32609544420b6f769d8988ffd38a2
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 4a23b4ae4c3050ab9ca2a829d04b270b700f322f1851ea5062bebaa6c91b7fa7
2025-08-18T09:12:52-04:00: Upload of new chunk 767541654c2b95fdedf38218bdbc7a1f7cf79d6efa4178defd21b03cea4d237c
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk ed7b809b75c3a76f6a4b39b432e0bd6cb6d3552c1e2928fb3527b487bb27b09a
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk f4839ed52566e55f579ec26ae5a574fff15636ea744e53053687388d20e54d15
2025-08-18T09:12:52-04:00: Upload of new chunk 2d7ab884622ee5c45418ea4a08aa6ca7f7d405b8202bc05cbad7bc40c2f3cec6
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk d2dbe7eff510910279d80f6879ed5fd318dbdc446f8b69f686203fb67645ebe9
2025-08-18T09:12:52-04:00: Upload of new chunk 5f14a9a6b60301a38b83fca661d416221d3d3aaaec3caf2150baeb4a93586b47
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 2d8392a32b6169b41d2ecff1913975cb0d086ac2570ecb79edf986b2ba396e79
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 9ffe76e1c8b23006a2c9d0229d846d42fcaa2b674190fb436193ae7acd98aca7
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk ea12e0038e8ce2c0b6556a64a89e72ba198134530d8fa6c0ac5595c937f956d9
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 07ccbd5ad8ee4042fa936eb480d0f0eb53bc16eeda38e13a18e3d365894af5fd
2025-08-18T09:12:52-04:00: Upload of new chunk 11b4f333a7affa5c0e2ae562f5261baf987f9879eb3f242be2ec20cd0c730fd1
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk b914fdaf19b8655ed79ae7ce18572c91eaa8a3232de8de4e69e37723e11ea2c9
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk e154eb92ddec56beebfd76aeb8bc99ebaab981d9c02d36d8457889829e4a0144
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk a46282ac7bc312e9d3b7c8e9c189575c540815b9137482269e96e335be8d1127
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk d0064f033b7b9c0988ff4b350d3710eb5917654c1f6655877a0fc3769273b501
2025-08-18T09:12:52-04:00: Upload of new chunk 836f46f2e8067ad9d463c22b09f28508a70935b210805ebb4f6f22c54306f542
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 09d2b3b76a2c87297945eee012a3d8530acd39c6f073dd61b46b4811469fb903
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk e681b87b1dd7fbad2b5c8315701296de1fe03d05ce7e802a9ffd8b8ba5e4bf32
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 07c9620146c8ab88220bd906e329216c629fb3b3fa0053962c7f4451909060a1
2025-08-18T09:12:52-04:00: Upload of new chunk 79d3e5e169bde489d51007462518b9319218fde8459d70e39127377bda72f396
2025-08-18T09:12:52-04:00: POST /dynamic_chunk
2025-08-18T09:12:52-04:00: Upload of new chunk 24fee820081e1b450d52043baaf94214d5a1b2a5d9d7b3ab56943dee4b54ddbe
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 31615550142de4343fec07ee628a6cc3ff27e996c58daef78dd34d272ef762ae
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 6613ad2fe7ded2a08d0c4e9d34a249a6af43fdd5316241f92df28319ba9b1546
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk cc920d0f9df8d3399d03a5b227a829b328a4870eef027955d60017e50c503efc
2025-08-18T09:12:53-04:00: Upload of new chunk 538aa64febb4cd6e8eb2a23fcb430740d50eca1c70188d48b8e738847d55a875
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 7fa7df4137e84cf595d543b9afa52bd517f2c1f7327ea213791607e1ceff8dc9
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 08c8da4bd7e2a01ae125d795ec671fc5bb4a69a6bd4b6c7a94f714ef8443ad12
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 7150e26d0163422172558b45cc322ac92e8dc4206b468fdc0088aa38f506fb96
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk ed0fad9a4e6d90ddbca445e518b0d153fd9ecc61483cafb6abb7180a54b9fd65
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk d0de1ccee759e5d483276ffaf407a30e0f81d53863b3960443ef2b3f10a62172
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk bd7a7e036f410a0d0787df9a2e6d58ed660fa5f7d4c70538d2e5e826fba111a4
2025-08-18T09:12:53-04:00: Upload of new chunk 8938161b7279ecdf5b1a027a5f4a968a965fb44f2c77317660d2eb9e11e72609
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk e7d987d73967d4876cf36e05f0c1f60ceaceefba5e0d7ed169e115ed3e39ed56
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk b963392855c013c63857455a445dd49b3370597e41f7e2d7322d28c1dc3d4809
2025-08-18T09:12:53-04:00: Upload of new chunk 3078f96a1f385afa9bf163a7870e7f98d6ea4fa6517e22bbc773bbc56f931847
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk f7ad0ec5dd4cba7c47e45ac078dd2b657d14e76a7a9d0ee509c4506dda2391d0
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk d529194228bdb3d4ca467bc1bf90bfd2c245ebd08620a5c18fb76bb244d0c5d8
2025-08-18T09:12:53-04:00: Upload of new chunk 8054cd722e960c1458c36f1835e8d2daa57bc240b518c43554e9a904bed8ce35
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk d0feadf67fd26468e65aadd03c0ed0a015ae6e0372e7bbd8a6de94bf67e6b6c4
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk ce48d03baafddc11b7907222ecc6846864f15d35c1c2f7de54027b26779fe929
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 87f578c29d69bbde2c821d8ccaf4b13020d991aabb9d9d6d9978c0b0dde760e3
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk da4eca65e00c21fef1c51151dc4dee06feae54f06d6d4fb469fd94e0647244c5
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk ed288384e38cfbc67e01dea9abb39c254871e667180e45e2962b70af5d0d5b49
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 18919676f4ebb719806a1b829b5cc7e71da96cf3cfcd1985dfa5608bcfbb3795
2025-08-18T09:12:53-04:00: Upload of new chunk eb80601793273e95ca659d777b88119e46486ecf5fbbd639c7144b7055f589f6
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 1b11d90b16307c5bbb361d3420cde6a20b3dbfa013ce24ca821170f711f0d01c
2025-08-18T09:12:53-04:00: Upload of new chunk 06c12135bdfb54dd8c423a9e7cb91a2b78b576fff911a7a11d7eec1e627f9c35
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 55e23cc2b4feea777f2297c77f26077b16ee53bb8b2485bfd750980e9508ff9d
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk f174da4ef6fdba6bf14e6aff6367a8f723ad0e67981fadaf82a1ed9b1752782a
2025-08-18T09:12:53-04:00: POST /dynamic_chunk
2025-08-18T09:12:53-04:00: Upload of new chunk 0a0ecf5b153f9914d3d4fc41c191d9b7ce2e0a6b50e473027e9dc8c902a873f5
2025-08-18T09:12:54-04:00: POST /dynamic_chunk: 400 Bad Request: failed to upload chunk to s3 backend
2025-08-18T09:12:54-04:00: POST /dynamic_chunk: 400 Bad Request: failed to upload chunk to s3 backend
2025-08-18T09:12:55-04:00: POST /dynamic_chunk: 400 Bad Request: failed to upload chunk to s3 backend
2025-08-18T09:12:55-04:00: backup ended and finish failed: backup ended but finished flag is not set.
2025-08-18T09:12:55-04:00: removing unfinished backup
2025-08-18T09:12:55-04:00: TASK ERROR: client error (Connect): Connection reset by peer (os error 104): Connection reset by peer (os error 104)

I also got the same thing when trying with an Hetzner S3 bucket as well.