PBS Corrupted chunk / S3 issue

N

Nxchu9123

New Member
Nov 26, 2024
6
2
3
Hello, I am currently on the latest version of PBS with an S3 backend on MEGA and a 1TB SSD for the cache. I performed several backups and when I started the verification, I only got failures:

syslog:
025-09-25T09:56:28.929438+02:00 backup proxmox-backup-proxy[664]: can't verify missing chunk with digest 4c2c657b9dfce1e6b82547119b53486774e11da1b1805e14681c1ca0f8fd7dc1
2025-09-25T09:56:29.569866+02:00 backup proxmox-backup-proxy[664]: TASK ERROR: verification failed - please check the log for details

...


pbs log:
2025-09-25T10:01:39+02:00: verify group s4_mega:ct/137 (1 snapshots)
2025-09-25T10:01:39+02:00: verify s4_mega:ct/137/2025-09-12T13:05:28Z
2025-09-25T10:01:39+02:00: check pct.conf.blob
2025-09-25T10:01:39+02:00: check root.pxar.didx
2025-09-25T10:18:30+02:00: "can't verify missing chunk with digest 4c2c657b9dfce1e6b82547119b53486774e11da1b1805e14681c1ca0f8fd7dc1"
2025-09-25T10:18:30+02:00: failed to copy corrupt chunk on s3 backend: 4c2c657b9dfce1e6b82547119b53486774e11da1b1805e14681c1ca0f8fd7dc1
2025-09-25T10:18:30+02:00: verified 1728.28/5022.01 MiB in 1010.41 seconds, speed 1.71/4.97 MiB/s (1 errors)


When I went to the datastore, then -> content -> more -> Refresh content from S3, my datastore goes into maintenance mode with the error:


2025-09-25T10:56:37+02:00: Fetching object ct/107/2025-09-12T11:34:27Z/index.json.blob
2025-09-25T10:56:37+02:00: Fetching object ct/107/2025-09-12T11:34:27Z/pct.conf.blob
2025-09-25T10:56:37+02:00: TASK ERROR: failed to download ct/107/2025-09-12T11:34:27Z/pct.conf.blob, not found


And my datastore is locked to "Maintenance mod s3 refresh". Here's my system information:
root@backup:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.1
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@backup:~#

proxmox-backup-server/stable,now 4.0.15-1 amd64 [installed]


On the S3 side, I can see some files (so the server is definitively writing), and when I do PBS backups, it says everything is fine...Any idea ?

Thanks you !
 
Last edited:
  • Like
Reactions: IloGus
Hello, I am currently on the latest version of PBS with an S3 backend on MEGA and a 1TB SSD for the cache. I performed several backups and when I started the verification, I only got failures:
Nxchu9123 said:
syslog:
025-09-25T09:56:28.929438+02:00 backup proxmox-backup-proxy[664]: can't verify missing chunk with digest 4c2c657b9dfce1e6b82547119b53486774e11da1b1805e14681c1ca0f8fd7dc1
2025-09-25T09:56:29.569866+02:00 backup proxmox-backup-proxy[664]: TASK ERROR: verification failed - please check the log for details

...


pbs log:
2025-09-25T10:01:39+02:00: verify group s4_mega:ct/137 (1 snapshots)
2025-09-25T10:01:39+02:00: verify s4_mega:ct/137/2025-09-12T13:05:28Z
2025-09-25T10:01:39+02:00: check pct.conf.blob
2025-09-25T10:01:39+02:00: check root.pxar.didx
2025-09-25T10:18:30+02:00: "can't verify missing chunk with digest 4c2c657b9dfce1e6b82547119b53486774e11da1b1805e14681c1ca0f8fd7dc1"
2025-09-25T10:18:30+02:00: failed to copy corrupt chunk on s3 backend: 4c2c657b9dfce1e6b82547119b53486774e11da1b1805e14681c1ca0f8fd7dc1
2025-09-25T10:18:30+02:00: verified 1728.28/5022.01 MiB in 1010.41 seconds, speed 1.71/4.97 MiB/s (1 errors)
Click to expand...
Seems like that snapshot is corrupt, for what you can try to do see https://forum.proxmox.com/threads/s3-buckets-constantly-failing-verification-jobs.169875/post-803887

Nxchu9123 said:
When I went to the datastore, then -> content -> more -> Refresh content from S3, my datastore goes into maintenance mode with the error:


2025-09-25T10:56:37+02:00: Fetching object ct/107/2025-09-12T11:34:27Z/index.json.blob
2025-09-25T10:56:37+02:00: Fetching object ct/107/2025-09-12T11:34:27Z/pct.conf.blob
2025-09-25T10:56:37+02:00: TASK ERROR: failed to download ct/107/2025-09-12T11:34:27Z/pct.conf.blob, not found
Click to expand...
This seems more like an issue on MEGA S4 site however.. The refresh lists the snapshots as reported by the MEGA API and then downloads them, and since you are in an s3 refresh maintenance mode, nothing else on the PBS side will remove objects in between. Can you verify that the object exists, e.g. via the providers tooling?

Nxchu9123 said:
And my datastore is locked to "Maintenance mod s3 refresh". Here's my system information:
root@backup:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.1
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@backup:~#

proxmox-backup-server/stable,now 4.0.15-1 amd64 [installed]


Any idea ?
Click to expand...
You can manually leave the maintenance mode again by clearing it in <datastore> > Options > Maintenance Mode > Edit. Then try the refresh again.
 
Chris said:
Hello, I am currently on the latest version of PBS with an S3 backend on MEGA and a 1TB SSD for the cache. I performed several backups and when I started the verification, I only got failures:

Seems like that snapshot is corrupt, for what you can try to do see https://forum.proxmox.com/threads/s3-buckets-constantly-failing-verification-jobs.169875/post-803887


This seems more like an issue on MEGA S4 site however.. The refresh lists the snapshots as reported by the MEGA API and then downloads them, and since you are in an s3 refresh maintenance mode, nothing else on the PBS side will remove objects in between. Can you verify that the object exists, e.g. via the providers tooling?


You can manually leave the maintenance mode again by clearing it in <datastore> > Options > Maintenance Mode > Edit. Then try the refresh again.
Click to expand...
Hello Chris,

Thanks you for your reply :) ! On Mega I can clearly see the pct.conf.blob files in the right folder. For the current maintenance mode, I've already tried to disable it but PBS reactivates it on its own. I'll take a look at the thread you mentioned but for now I'm going to start fresh by deleting my datastore to see if that changes anything (I don't mind, it's not production data).

I'll give an update.
 
Small update, now the verification is working but when i click on refresh from s3 I got the error:
2025-09-25T11:54:30+02:00: TASK ERROR: failed to download ct/137/2025-09-25T09:41:28Z/client.log.blob, not found depsite its present on the s3.

Not sure if this is normal but i've multiple directory for only one backup..

Same issue on VM i think this is causing the issue, mind I ask your opinion @Chris ?
1758794170367.png
 
Last edited:
Nxchu9123 said:
Small update, now the verification is working but when i click on refresh from s3 I got the error:
2025-09-25T11:54:30+02:00: TASK ERROR: failed to download ct/137/2025-09-25T09:41:28Z/client.log.blob, not found depsite its present on the s3.
Click to expand...
Okay, thanks for the report. Please open an issue for this on https://bugzilla.proxmox.com, referencing this thread so we can look into it. At first glance I suspect the URI encoding for the S3 object keys might lead to the incompatibility.

Nxchu9123 said:
Not sure if this is normal but i've multiple directory for only one backup..

Same issue on VM i think this is causing the issue, mind I ask your opinion @Chris ?
Click to expand...
I think this is just how MEGA shows you the object prefixes (note S3 does not directly have a folder structure), not grouping identical ones for each object? Can you check that you have 8 objects in that key sub-prefix?
 
Chris said:
Okay, thanks for the report. Please open an issue for this on https://bugzilla.proxmox.com, referencing this thread so we can look into it. At first glance I suspect the URI encoding for the S3 object keys might lead to the incompatibility.
Click to expand...
I created the bug report!

> I think this is just how MEGA shows you the object prefixes (note S3 does not directly have a folder structure), not grouping identical ones for each object? Can you check that you have 8 objects in that key sub-prefix?

Unfortunately, I have already emptied the datastore and I have gone back to using rclone for my backup replication. I don't know if this is useful, but on Mega I was able to move each object individually (so they were properly treated as individual objects); they even have different sizes (surprisingly very small).
 
Definitively a Mega s4 issue, same error using rclone..
ERROR : ct/135/2025-09-25T11:13:21Z/pct.conf.blob: Failed to set modification time: object not found
 
Last edited:
You must log in or register to reply here.
Top Bottom