Inconsistent info about validation status

R

rahman

Renowned Member
Nov 1, 2010
93
5
73
Hi,

I have a Ceph Backed S3 datastore that is synced from local datastore. I am running weekly verify jobs. The problem is last job complains about validation errors but when I look content of datastore on PBS GUI it does not match:

Code: 
Job ID:    v-35b45b04-d2ac
Datastore: ulakfkm

Verification failed on these snapshots/groups:

    ns/default/vm/163/2025-07-05T05:25:58Z
    ns/default/vm/164/2026-03-29T22:00:44Z
    ns/default/vm/183/2025-01-07T06:00:36Z
    ns/default/vm/206/2026-03-13T18:01:18Z
    ns/default/vm/206/2026-03-03T18:01:12Z


Please visit the web interface for further details:

1775044235162.png

As you can see in GUI vm/163/2025-07-05T05:25:58Z snapshot shows its validated.


1775044393209.png

vm/164/2026-03-29T22:00:44Z snapshot is not shown as failed.


1775044649547.png

For vm/206/ only one of the snapshots shown as failed, the other is shown as "ok".

So what should be the cause of this?

I also run daily GC jobs, daily sync jobs (pull from local), prune job 4 hour interval. Weekly verify job took about 2-3 days. Is it safe if GC, sync and prune jobs to run when verify job still running?

Regards,
Rahman
 

Attachments

Hi,
please provide the output of proxmox-backup-manager version --verbose.

First guess is that likely because the verification failed to download the chunk due to a transient network issue. But the verify state should be updated for that case as well. Could you also check the systemd journal for any s3 client related error messages around the time the verification is running and check if the snapshots are okay if verified manually?
 
Code: 
root@pbs1:~# proxmox-backup-manager version --verbose
proxmox-backup                      4.0.0         running kernel: 6.17.13-2-pve
proxmox-backup-server               4.1.5-2       running version: 4.1.4
proxmox-kernel-helper               9.0.4
proxmox-kernel-6.17                 6.17.13-2
proxmox-kernel-6.17.13-2-pve-signed 6.17.13-2
proxmox-kernel-6.17.4-2-pve-signed  6.17.4-2
proxmox-kernel-6.17.2-1-pve-signed  6.17.2-1
ifupdown2                           3.3.0-1+pmx12
libjs-extjs                         7.0.0-5
proxmox-backup-docs                 4.1.5-1
proxmox-backup-client               4.1.5-1
proxmox-mail-forward                1.0.2
proxmox-mini-journalreader          1.6
proxmox-offline-mirror-helper       0.7.3
proxmox-widget-toolkit              5.1.8
pve-xtermjs                         5.5.0-3
smartmontools                       7.4-pve1
zfsutils-linux                      2.4.1-pve1

Here is the journal logs for this snapshot:

Code: 
2026-03-30T12:23:48+03:00: verify ulakfkm:vm/183/2025-01-07T06:00:36Z
2026-03-30T12:23:48+03:00:   check qemu-server.conf.blob
2026-03-30T12:23:48+03:00:   check drive-scsi1.img.fidx
2026-03-30T13:39:36+03:00:   verified 271680.08/413196.00 MiB in 4548.00 seconds, speed 59.74/90.85 MiB/s (6 errors)
2026-03-30T13:39:36+03:00: verify ulakfkm:vm/183/2025-01-07T06:00:36Z/drive-scsi1.img.fidx failed: chunks could not be verified

Code: 
root@pbs1:~# journalctl --since "2026-03-30T12:23:48+03:00" --until "2026-03-30T13:39:36+03:00"
Mar 30 12:41:06 pbs1 smartd[1891]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 67 to 68
Mar 30 12:41:06 pbs1 smartd[1891]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 33 to 32
Mar 30 12:43:38 pbs1 proxmox-backup-proxy[2299]: rrd journal successfully committed (33 files in 0.034 seconds)
Mar 30 13:11:06 pbs1 smartd[1891]: Device: /dev/sdg [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 67
Mar 30 13:11:06 pbs1 smartd[1891]: Device: /dev/sdg [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 32 to 33
Mar 30 13:13:38 pbs1 proxmox-backup-proxy[2299]: rrd journal successfully committed (33 files in 0.032 seconds)
Mar 30 13:17:01 pbs1 CRON[3147724]: pam_unix(cron:session): session opened for user root(uid=0) by root(uid=0)
Mar 30 13:17:01 pbs1 CRON[3147726]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)
Mar 30 13:17:01 pbs1 CRON[3147724]: pam_unix(cron:session): session closed for user root
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

I will run a verify job for vm/183 snapshot for test.

Also any feedback about this subject:
I also run daily GC jobs, daily sync jobs (pull from local), prune job 4 hour interval. Weekly verify job took about 2-3 days. Is it safe if GC, sync and prune jobs to run when verify job still running?
Click to expand...
 
rahman said:
Mar 30 13:17:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Click to expand...
Okay, so as expected there is a transient network error, the verification task fails due to this, but it is not possible to determine if the snapshot is okay or corrupt from this. Will check if we can improve the verify state handling here.

rahman said:
I also run daily GC jobs, daily sync jobs (pull from local), prune job 4 hour interval. Weekly verify job took about 2-3 days. Is it safe if GC, sync and prune jobs to run when verify job still running?
Click to expand...
Yes, although I would not recommend to many operations at the same time. The more tasks are run in parallel, the more requests are being send at the same time to the S3 endpoint API. Could be that you are seeing above connection errors because of this.
 
Chris said:
Yes, although I would not recommend to many operations at the same time. The more tasks are run in parallel, the more requests are being send at the same time to the S3 endpoint API. Could be that you are seeing above connection errors because of this.
Click to expand...

As verify takes so long, GC and sync jobs runs once a day while verify also running. Can't know if it causes this. But I checked sync and GC jobs for "2026-03-30T12:23:48+03:00" to "2026-03-30T13:39:36+03:00" which is vm/183 verify errored, and there is was no gc or sync jobs running between these times. GC job scheduled to run every midnight , and sync job every night at 03:00, both Turkey timezone (+03:00).
 
Code: 
2026-04-01T16:11:38+03:00: verify ulakfkm:vm/206/2026-03-03T18:01:12Z
2026-04-01T16:11:38+03:00:   check qemu-server.conf.blob
2026-04-01T16:11:38+03:00:   check drive-scsi0.img.fidx
2026-04-01T16:32:48+03:00:   verified 23359.43/50996.00 MiB in 1270.29 seconds, speed 18.39/40.15 MiB/s (0 errors)
2026-04-01T16:32:49+03:00: TASK OK

Code: 
2026-04-01T16:43:07+03:00: verify ulakfkm:vm/183/2025-01-07T06:00:36Z
2026-04-01T16:43:07+03:00:   check qemu-server.conf.blob
2026-04-01T16:43:07+03:00:   check drive-scsi1.img.fidx
2026-04-01T21:18:19+03:00:   verified 271704.08/413232.00 MiB in 16511.23 seconds, speed 16.46/25.03 MiB/s (0 errors)
2026-04-01T21:18:19+03:00:   check drive-scsi0.img.fidx
2026-04-02T02:28:29+03:00:   verified 166003.01/1013796.00 MiB in 18610.11 seconds, speed 8.92/54.48 MiB/s (0 errors)
2026-04-02T02:28:29+03:00: TASK OK

I tested to manually verify two of the failed snapshots and they validated without any problem.
 
Last edited:
Hi, still having same issues with verification jobs. I tried removing rate limiting for S3 endpoint in PBS S3 endpoint settings, did not make any difference. Added static mapping to /etc/hosts file for S3 endpoint IP address, did not worked either.

So is there a way to make PBS S3 client more resilient to random hiccups? Some settings so it just retries to process failed chunk for configured times?

Here is the latest job logs:

Code: 
Job ID:    v-35b45b04-d2ac
Datastore: ulakfkm

Verification failed on these snapshots/groups:

    ns/default/vm/113/2026-04-04T04:02:46Z
    ns/default/vm/137/2026-04-04T04:10:47Z
    ns/default/vm/147/2026-04-12T04:03:34Z
    ns/default/vm/149/2026-04-12T04:05:45Z
    ns/default/vm/190/2026-04-16T18:00:02Z
    ns/default/vm/190/2026-04-15T18:00:09Z
    ns/default/vm/190/2026-04-13T18:00:00Z
    ns/default/vm/190/2026-04-10T18:00:02Z
    ns/default/vm/268/2026-04-17T16:01:05Z
    ns/default/vm/268/2026-02-27T18:01:34Z


Please visit the web interface for further details:

2026-04-18T03:41:22+03:00: verify ulakfkm:vm/113/2026-04-04T04:02:46Z
2026-04-18T03:41:22+03:00:   check qemu-server.conf.blob
2026-04-18T03:41:22+03:00:   check drive-scsi0.img.fidx
2026-04-18T08:18:43+03:00:   verified 993463.89/1048244.00 MiB in 16640.90 seconds, speed 59.70/62.99 MiB/s (7 errors)
2026-04-18T08:18:43+03:00: verify ulakfkm:vm/113/2026-04-04T04:02:46Z/drive-scsi0.img.fidx failed: chunks could not be verified
2026-04-18T08:18:43+03:00: percentage done: 5.68% (7/129 groups, 1/3 snapshots in group #8)

root@pbs1:~# journalctl --since "2026-04-18T03:41:22+03:00" --until "2026-04-18T08:18:43+03:00" --no-pager | grep failed
Apr 18 04:19:20 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Apr 18 04:21:14 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 04:21:19 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 04:21:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 04:21:35 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 04:23:19 pbs1 proxmox-backup-proxy[2299]: TASK ERROR: sync failed with some errors.
Apr 18 05:26:17 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection
Apr 18 08:07:51 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection



2026-04-18T11:25:21+03:00: verify ulakfkm:vm/137/2026-04-04T04:10:47Z
2026-04-18T11:25:21+03:00:   check qemu-server.conf.blob
2026-04-18T11:25:21+03:00:   check drive-scsi0.img.fidx
2026-04-18T13:08:09+03:00:   verified 326071.54/449960.00 MiB in 6167.57 seconds, speed 52.87/72.96 MiB/s (78 errors)
2026-04-18T13:08:09+03:00: verify ulakfkm:vm/137/2026-04-04T04:10:47Z/drive-scsi0.img.fidx failed: chunks could not be verified
2026-04-18T13:08:09+03:00: percentage done: 24.29% (31/129 groups, 1/3 snapshots in group #32)

root@pbs1:~# journalctl --since "2026-04-18T11:25:21+03:00" --until "2026-04-18T13:08:09+03:00" --no-pager | grep failed
Apr 18 12:02:32 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:03:33 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:03:37 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:04:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:04:50 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:05:19 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:05:25 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:01 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:03 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:07 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:38 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:51 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:07:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:08:12 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:08:23 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:08:29 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:08:57 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:33 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:40 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:42 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:43 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:09:48 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:10:23 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:16:33 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:16:34 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:17:12 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:19:36 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:19:44 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:19:48 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:21:53 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:22:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:22:19 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:22:32 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:22:52 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:23:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:26:13 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:28:29 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:28:53 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:29:03 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:29:08 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:29:52 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:30:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:32:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:33:12 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:33:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:34:05 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:34:09 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:34:54 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:34:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:35:16 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:35:54 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:35:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:35:56 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:36:04 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:36:09 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:36:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:36:48 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:36:56 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:37:22 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:37:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:37:33 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:37:50 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:38:04 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:38:13 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:38:52 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:38:53 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:40:16 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:40:56 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:40:57 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:41:16 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:41:17 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:41:42 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:43:00 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:48:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 18 12:49:04 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
root@pbs1:~#






2026-04-18T23:53:35+03:00: verify ulakfkm:vm/149/2026-04-12T04:05:45Z
2026-04-18T23:53:35+03:00:   check qemu-server.conf.blob
2026-04-18T23:53:35+03:00:   check drive-scsi1.img.fidx
2026-04-19T04:10:07+03:00:   verified 915236.59/1102420.00 MiB in 15391.81 seconds, speed 59.46/71.62 MiB/s (11 errors)
2026-04-19T04:10:07+03:00: verify ulakfkm:vm/149/2026-04-12T04:05:45Z/drive-scsi1.img.fidx failed: chunks could not be verified

root@pbs1:~# journalctl --since "2026-04-18T23:53:35+03:00" --until "2026-04-19T04:10:07+03:00" --no-pager | grep failed
Apr 19 00:06:51 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:07:05 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:07:15 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:07:16 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:07:35 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:07:58 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:08:14 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:08:45 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:08:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:08:59 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 00:09:11 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (Connect)
root@pbs1:~#



2026-04-19T19:29:57+03:00: verify ulakfkm:vm/190/2026-04-16T18:00:02Z
2026-04-19T19:29:57+03:00:   check qemu-server.conf.blob
2026-04-19T19:29:57+03:00:   check drive-scsi1.img.fidx
2026-04-19T19:48:31+03:00:   verified 65677.37/66532.00 MiB in 1113.91 seconds, speed 58.96/59.73 MiB/s (1 errors)
2026-04-19T19:48:31+03:00: verify ulakfkm:vm/190/2026-04-16T18:00:02Z/drive-scsi1.img.fidx failed: chunks could not be verified
2026-04-19T19:48:31+03:00:   check drive-scsi0.img.fidx
2026-04-19T19:56:31+03:00:   verified 27826.41/106128.00 MiB in 480.46 seconds, speed 57.92/220.89 MiB/s (2 errors)
2026-04-19T19:56:31+03:00: verify ulakfkm:vm/190/2026-04-16T18:00:02Z/drive-scsi0.img.fidx failed: chunks could not be verified

root@pbs1:~# journalctl --since "2026-04-19T19:29:57+03:00" --until "2026-04-19T19:56:31+03:00" --no-pager | grep failed
Apr 19 19:48:21 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 19:50:04 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
Apr 19 19:50:07 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)
root@pbs1:~#
 
Another snapshot that failed:

Code: 
2026-04-18T14:33:54+03:00: verify ulakfkm:vm/147/2026-04-12T04:03:34Z

2026-04-18T14:33:54+03:00:   check qemu-server.conf.blob

2026-04-18T14:33:54+03:00:   check drive-scsi1.img.fidx

2026-04-18T23:37:37+03:00:   verified 1929315.07/2029084.00 MiB in 32622.67 seconds, speed 59.14/62.20 MiB/s (107 errors)

2026-04-18T23:37:37+03:00: verify ulakfkm:vm/147/2026-04-12T04:03:34Z/drive-scsi1.img.fidx failed: chunks could not be verified



root@pbs1:~# journalctl --since "2026-04-18T14:33:54+03:00" --until "2026-04-18T23:37:37+03:00" --no-pager | grep failed

Apr 18 17:08:10 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 17:08:14 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 17:08:14 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 17:08:22 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 17:08:26 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 17:08:47 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:04:56 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (Connect)

Apr 18 20:04:56 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (Connect)

Apr 18 20:04:57 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (Connect)

Apr 18 20:05:05 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:05:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:06:20 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:06:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:06:29 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:06:48 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:07:16 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:08:22 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:08:33 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:09:06 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:09:54 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:11:45 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:11:52 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:12:36 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:17:15 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:17:21 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:18:31 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:18:38 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:19:31 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:35:23 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:35:44 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:35:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:37:24 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:37:28 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:37:44 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:38:01 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:38:06 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:38:08 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:38:23 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:38:32 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:38:57 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:39:37 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:39:38 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:42:04 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:43:01 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:43:19 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:45:15 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:45:32 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:45:45 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:49:40 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:49:47 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:49:52 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:54:31 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:55:06 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:55:12 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:55:16 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:55:55 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:56:04 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 20:56:14 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:05:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:05:34 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (Connect)

Apr 18 21:08:22 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:08:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:10:13 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:11:03 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:11:07 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:11:07 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:11:26 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:11:30 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:12:38 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:13:17 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:13:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:13:31 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:15:19 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:15:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:15:38 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:15:59 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:16:24 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

Apr 18 21:17:06 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:17:38 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:18:59 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:19:01 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:41:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

Apr 18 21:41:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

Apr 18 21:41:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

Apr 18 21:41:27 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

Apr 18 21:41:43 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - error reading a body from connection

Apr 18 21:42:51 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:54:51 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:56:14 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 21:58:50 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:02:18 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:03:00 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:05:20 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:10:17 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:10:22 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:13:58 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:14:35 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:18:39 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:23:20 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:23:21 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:23:50 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:31:53 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:43:12 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:45:03 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:45:13 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:45:31 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

Apr 18 22:45:35 pbs1 proxmox-backup-proxy[2299]: can't verify chunk, load failed - client error (SendRequest)

root@pbs1:~#
 
You must log in or register to reply here.
Top Bottom
Back