That's not a good choice for a local cache... i would recommended to use local fast nvme ssds, best with some redundancy for error correction.
Proxmox Backup Server 4.0 released!
- Thread starter t.lamprecht
- Start date
Allow me a follow up question regarding the local cache. What kind of data does the cache hold and how long does it cache these files? I see a significant amount of data still stored in that cache. Is there any documentation how the cache is used or for it is used for in detail?
Thanks again for your help.
Thanks again for your help.
The cache stores snapshot metadata such as index file, owner files, ecc. as well as backup data chunks. Chunks are stored in a least recently used cache, evicting older ones if no more slots are available. The cache will use as much storage space as available, therefore the recommendation to use a dedicated disk, partition or dataset with quota. Some more considerations are also to be found in the docs https://pbs.proxmox.com/docs/storage.html#datastores-with-s3-backend
the larger backups (600G) are failing still after updating to 4.0.15
Its interesting to note the network graph for the backup server vs smaller backups, only incoming traffic , none out to the s3 based storage
are there any logs I can view to try get a clue - checked the journal log and nothing stands out like the previous vpn network issue (we use wireguard)
Edit: I wonder if there are any files from the failed jobs causing issues on the cache disk or s3 storage
Edit2: just backed up a 1.2TB VM - worked fine, leads me to believe either some files are stuck in s3 or cache related to that VMid that fails or something on the VM is corrupt, but it stopped working after the VPN network backup proxy issue
Its interesting to note the network graph for the backup server vs smaller backups, only incoming traffic , none out to the s3 based storage
are there any logs I can view to try get a clue - checked the journal log and nothing stands out like the previous vpn network issue (we use wireguard)
Edit: I wonder if there are any files from the failed jobs causing issues on the cache disk or s3 storage
Edit2: just backed up a 1.2TB VM - worked fine, leads me to believe either some files are stuck in s3 or cache related to that VMid that fails or something on the VM is corrupt, but it stopped working after the VPN network backup proxy issue
Attachments
Last edited:
Hi!
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6750
What exact error message do you get for the failed backups? Anything of interest in the PVE task log or the corresponding PBS backup task log?
We discovered an issue leading to possible deadlocks. A fix is work in progress, please subscribe to the issue [0] for status updates.
[0] https://bugzilla.proxmox.com/show_bug.cgi?id=6750
Last edited:
Code:
INFO: 98% (818.4 GiB of 835.1 GiB) in 1h 38m 34s, read: 149.0 MiB/s, write: 147.0 MiB/s
INFO: 99% (826.8 GiB of 835.1 GiB) in 1h 39m 32s, read: 147.5 MiB/s, write: 145.1 MiB/s
INFO: 100% (835.1 GiB of 835.1 GiB) in 1h 40m 33s, read: 139.6 MiB/s, write: 136.7 MiB/s
ERROR: backup close image failed: command error: broken pipe
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 108 failed - backup close image failed: command error: broken pipe
INFO: Failed at 2025-09-28 04:43:41
INFO: Backup job finished with errors
TASK ERROR: job errorsI do have a dirty bitmap on the large volume due to the backups to the old backup server which I since removed, but I assume the dirty bit doesnt go away as the backup for that volume never completes?
PBS log looks interesting:
Code:
2025-09-27T17:10:38+02:00: starting new backup on datastore 'hetzners3ds' from ::ffff:192.168.122.15: "vm/108/2025-09-27T15:10:37Z"
2025-09-27T17:10:38+02:00: GET /previous: 400 Bad Request: no valid previous backup
2025-09-27T17:10:38+02:00: created new fixed index 1 ("vm/108/2025-09-27T15:10:37Z/drive-scsi0.img.fidx")
2025-09-27T17:10:38+02:00: Skip upload of already encountered chunk bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
2025-09-27T17:10:38+02:00: created new fixed index 2 ("vm/108/2025-09-27T15:10:37Z/drive-scsi1.img.fidx")
2025-09-27T17:10:38+02:00: Skip upload of already encountered chunk bb9f8df61474d25e71fa00722318cd387396ca1736605e1248821cc0de3d3af8
2025-09-27T17:10:39+02:00: Uploaded blob to object store: .cnt/vm/108/2025-09-27T15:10:37Z/qemu-server.conf.blob
2025-09-27T17:10:39+02:00: add blob "/mnt/datastore/s3cache/vm/108/2025-09-27T15:10:37Z/qemu-server.conf.blob" (1081 bytes, comp: 1081)
2025-09-27T17:10:39+02:00: Upload of new chunk 7cea0197776e28a0558513e967af200d4c6b073abf57dc702e37967b2f603e25
2025-09-27T17:10:39+02:00: Skip upload of already encountered chunk 1061b7ee267fabb1e4e61d2d752c4ec92c1c6144c34a95b3dae9df742b4fffdd
2025-09-27T17:10:39+02:00: Caching of chunk 7cea0197776e28a0558513e967af200d4c6b073abf57dc702e37967b2f603e25
2025-09-27T17:10:39+02:00: Skip upload of already encountered chunk 59aed2a3f720cfb9d58fae695d52cb8a6512a74e9bfbe7158cdd88718160bc9c
2025-09-27T17:10:39+02:00: Skip upload of already encountered chunk 366ce253154ab63930d9787323347ed7b92b3361c966bd1ac93f39043e3d4941
2025-09-27T17:10:39+02:00: Skip upload of already encountered chunk fe2ff82c2a214a6d28e70e716c1f297a52361d6b47abe285a78c5b054fb958d8
removed similar entries
2025-09-27T17:10:42+02:00: Caching of chunk 28e2eeae5799e1d3f610f0c8ed7bd216a98fd228a7a97da1b7e5f1efe2f70a2d
2025-09-27T17:10:42+02:00: Upload of new chunk c226a367c733d50411847e9b29370168b958300cac5ddf2528edc2a34c13d77b
2025-09-27T17:10:42+02:00: Upload of new chunk 9d8a5a0b10feff4a26c02ce9e56cb0c704d71805a61849dd4e45607dbb3ed744
2025-09-27T17:10:42+02:00: Caching of chunk 12b7b3157d0c8b292e012a7027a62a0b6807fc75b8f5525044d92adb5c08463e
2025-09-27T17:10:42+02:00: Caching of chunk ba5b40fa07c88e152e2e24cddfa3dfc0a191c48dcf4e858d8ae6daded5529f53
removed similar entries
2025-09-27T18:52:45+02:00: Skip upload of already encountered chunk 3bad1bf6e5b57cb8bba3eb30d0ff1418b1f2294322648e8a16c31d5965c8607d
2025-09-27T18:52:45+02:00: Upload of new chunk 19a596026fc5d92de48a814285b6114c85ffa8839e5302bfb29b3df5c2cb4044
2025-09-27T18:52:45+02:00: Upload of new chunk b743eb6750f175fd852343c4bc9bfb375e19c73c73027ed8f2bda9b32b0e1f39
2025-09-27T18:52:45+02:00: Caching of chunk 19a596026fc5d92de48a814285b6114c85ffa8839e5302bfb29b3df5c2cb4044
2025-09-27T18:52:45+02:00: POST /fixed_close: 400 Bad Request: Atomic rename file "/mnt/datastore/s3cache/vm/108/2025-09-27T15:10:37Z/drive-scsi1.img.fidx" failed - No such file or directory (os e
rror 2)
2025-09-27T18:52:45+02:00: Upload of new chunk 6096a2f7554ae3ebfbf2766f2f939496e79a4f034ea00613f97af7021873f77b
2025-09-27T18:52:45+02:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection
2025-09-27T18:52:45+02:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection
2025-09-27T18:52:45+02:00: POST /fixed_chunk: 400 Bad Request: error reading a body from connection
2025-09-27T18:52:45+02:00: backup failed: connection error
2025-09-27T18:52:45+02:00: removing failed backup
2025-09-27T18:52:45+02:00: removing backup snapshot "/mnt/datastore/s3cache/vm/108/2025-09-27T15:10:37Z"
2025-09-27T18:52:45+02:00: TASK ERROR: connection error: bytes remaining on stream