PBS: Newly created Backups are marked as corrupt

[wagnbeu0]

I am Using Proxmox 8 on a Lenovo Server. Locally is a 1 TB SSD installed where my Proxmox Installation and Backups are stored. I´m running a PBS in a LXC.
The PBS has connected a 5 TB NFS Share as Backup Volume and also Volume where the CT are stored and running..

Backups are running and created as they should, no error.

But when I verify my backups, I can see that almost all of them are marked as "Corrupt":

2025-05-27T11:22:20+02:00: verify datastore Backup
2025-05-27T11:22:20+02:00: found 14 groups
2025-05-27T11:22:20+02:00: verify group Backup:ct/100 (2 snapshots)
2025-05-27T11:22:20+02:00: SKIPPED: verify Backup:ct/100/2025-05-12T15:49:18Z (recently verified)
2025-05-27T11:22:20+02:00: percentage done: 3.57% (0/14 groups, 1/2 snapshots in group #1)
2025-05-27T11:22:20+02:00: SKIPPED: verify Backup:ct/100/2025-05-12T15:33:55Z (recently verified)
2025-05-27T11:22:20+02:00: percentage done: 7.14% (1/14 groups)
2025-05-27T11:22:20+02:00: verify group Backup:ct/103 (28 snapshots)
2025-05-27T11:22:20+02:00: verify Backup:ct/103/2025-05-27T09:00:10Z
2025-05-27T11:22:20+02:00: check pct.conf.blob
2025-05-27T11:22:20+02:00: check root.pxar.didx
2025-05-27T11:25:56+02:00: can't verify chunk, load failed - store 'Backup', unable to load chunk '6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb' - No such file or directory (os error 2)
2025-05-27T11:25:56+02:00: verified 6795.56/15911.90 MiB in 215.26 seconds, speed 31.57/73.92 MiB/s (1 errors)
2025-05-27T11:25:56+02:00: verify Backup:ct/103/2025-05-27T09:00:10Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:25:56+02:00: check catalog.pcat1.didx
2025-05-27T11:25:56+02:00: verified 2.13/5.81 MiB in 0.40 seconds, speed 5.36/14.62 MiB/s (0 errors)
2025-05-27T11:25:56+02:00: percentage done: 7.40% (1/14 groups, 1/28 snapshots in group #2)
2025-05-27T11:25:56+02:00: verify Backup:ct/103/2025-05-27T08:00:08Z
2025-05-27T11:25:56+02:00: check pct.conf.blob
2025-05-27T11:25:56+02:00: check root.pxar.didx
...
2025-05-27T11:26:06+02:00: check pct.conf.blob
2025-05-27T11:26:06+02:00: check root.pxar.didx
2025-05-27T11:26:06+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:08+02:00: verified 46.74/210.97 MiB in 1.67 seconds, speed 28.07/126.69 MiB/s (1 errors)
2025-05-27T11:26:08+02:00: verify Backup:ct/103/2025-05-27T04:00:05Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:26:08+02:00: check catalog.pcat1.didx
2025-05-27T11:26:08+02:00: verified 0.67/1.91 MiB in 0.14 seconds, speed 4.83/13.87 MiB/s (0 errors)
2025-05-27T11:26:08+02:00: percentage done: 8.67% (1/14 groups, 6/28 snapshots in group #2)
2025-05-27T11:26:08+02:00: verify Backup:ct/103/2025-05-27T03:00:04Z
2025-05-27T11:26:08+02:00: check pct.conf.blob
2025-05-27T11:26:08+02:00: check root.pxar.didx
2025-05-27T11:26:08+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:10+02:00: verified 53.52/256.18 MiB in 1.98 seconds, speed 27.03/129.37 MiB/s (1 errors)
2025-05-27T11:26:10+02:00: verify Backup:ct/103/2025-05-27T03:00:04Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:26:10+02:00: check catalog.pcat1.didx
2025-05-27T11:26:10+02:00: verified 0.80/2.19 MiB in 0.18 seconds, speed 4.50/12.28 MiB/s (0 errors)
2025-05-27T11:26:10+02:00: percentage done: 8.93% (1/14 groups, 7/28 snapshots in group #2)
2025-05-27T11:26:10+02:00: verify Backup:ct/103/2025-05-27T02:03:20Z
2025-05-27T11:26:10+02:00: check pct.conf.blob
2025-05-27T11:26:10+02:00: check root.pxar.didx
2025-05-27T11:26:10+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:12+02:00: verified 35.73/123.25 MiB in 1.27 seconds, speed 28.14/97.08 MiB/s (1 errors)
2025-05-27T11:26:12+02:00: verify Backup:ct/103/2025-05-27T02:03:20Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:26:12+02:00: check catalog.pcat1.didx
2025-05-27T11:26:12+02:00: verified 0.67/1.92 MiB in 0.10 seconds, speed 6.77/19.39 MiB/s (0 errors)
2025-05-27T11:26:12+02:00: percentage done: 9.18% (1/14 groups, 8/28 snapshots in group #2)
2025-05-27T11:26:12+02:00: verify Backup:ct/103/2025-05-27T01:00:03Z
2025-05-27T11:26:12+02:00: check pct.conf.blob
2025-05-27T11:26:12+02:00: check root.pxar.didx
2025-05-27T11:26:12+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:16+02:00: verified 77.07/353.31 MiB in 4.22 seconds, speed 18.26/83.73 MiB/s (1 errors)
2025-05-27T11:26:16+02:00: verify Backup:ct/103/2025-05-27T01:00:03Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:26:16+02:00: check catalog.pcat1.didx
2025-05-27T11:26:16+02:00: verified 0.97/2.74 MiB in 0.22 seconds, speed 4.47/12.67 MiB/s (0 errors)
2025-05-27T11:26:16+02:00: percentage done: 9.44% (1/14 groups, 9/28 snapshots in group #2)
2025-05-27T11:26:16+02:00: verify Backup:ct/103/2025-05-27T00:00:03Z
2025-05-27T11:26:16+02:00: check pct.conf.blob
2025-05-27T11:26:16+02:00: check root.pxar.didx
2025-05-27T11:26:16+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:18+02:00: verified 57.32/222.91 MiB in 1.98 seconds, speed 28.93/112.50 MiB/s (1 errors)
2025-05-27T11:26:18+02:00: verify Backup:ct/103/2025-05-27T00:00:03Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:26:18+02:00: check catalog.pcat1.didx
2025-05-27T11:26:18+02:00: verified 1.11/3.02 MiB in 0.19 seconds, speed 5.87/16.03 MiB/s (0 errors)
2025-05-27T11:26:18+02:00: percentage done: 9.69% (1/14 groups, 10/28 snapshots in group #2)
2025-05-27T11:26:18+02:00: verify Backup:ct/103/2025-05-26T23:00:03Z
2025-05-27T11:26:18+02:00: check pct.conf.blob
2025-05-27T11:26:18+02:00: check root.pxar.didx
2025-05-27T11:26:19+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:20+02:00: verified 32.97/122.92 MiB in 1.06 seconds, speed 31.23/116.45 MiB/s (1 errors)
2025-05-27T11:26:20+02:00: verify Backup:ct/103/2025-05-26T23:00:03Z/root.pxar.didx failed: chunks could not be verified
2025-05-27T11:26:20+02:00: check catalog.pcat1.didx
2025-05-27T11:26:20+02:00: verified 0.67/1.91 MiB in 0.04 seconds, speed 15.59/44.79 MiB/s (0 errors)
2025-05-27T11:26:20+02:00: percentage done: 9.95% (1/14 groups, 11/28 snapshots in group #2)
2025-05-27T11:26:20+02:00: verify Backup:ct/103/2025-05-26T22:00:04Z
2025-05-27T11:26:20+02:00: check pct.conf.blob
2025-05-27T11:26:20+02:00: check root.pxar.didx
2025-05-27T11:26:20+02:00: chunk 6190160a9799def2d9c84ed618ff4425830d6cbdffcc16b37db4204099535cdb was marked as corrupt
2025-05-27T11:26:24+02:00: can't verify chunk, load failed - store 'Backup', unable to load chunk '64676055f52c2a3059350646bff2e587ff1a95bbe6f19da83a3d9fd7b6b99726' - No such file or directory (os error 2)


What can be the problem that newly created backups cannot be verified? The NFS Server is running EXT4 Filesystem.

 

[Chris]

I suggest you re-verify all the snapshots in your datastore, as you have:

1. missing chunks
2. corrupt chunks

The newly created backup snapshots containing corrupt chunks most likely because they re-used a known chunk, which only after (by the verify job) has been detected as corrupt. The next backup will however not reuse such chunks (as the snapshot is now marked as verify failed) and re-upload the chunk. So if the data is unchanged on the source, the chunk is re-generated and re-uploaded, so this will heal you snapshots.