Backups keep failing verification

Solus

Member
Sep 18, 2024
2
0
6
I'm trying to get PBS working smoothly but I keep running into a snag. I'm able to backup the VM/CT's fine, but it occurs often that the backup can't be verified. I've tried removing the backups and taking new ones with no change, they keep failing. One just keeps failing the initial backup verification.

I read that it could be corrupt VM disks, so I cloned the disk to a second on I added, all on the same VM, removed the old disk and tried then. It appeared to have worked, at least on the initial backup. Then, subsequent backups after that initial one just keeps failing to be verified.

I'm at a loss on what I should do, is my Proxmox VE nodes (there are two) corrupted in some way? (It wouldn't surprise me I guess, I messed with it a lot when setting it up)
1795028818094.png

1795028844421.png

Let me know what information you need from me to help understand what's going on! I'd like to learn what I'm doing wrong so I can prevent it in the future...

This is what I get for that task...

Code:
2026-11-18T13:37:35-05:00: verify External:vm/102/2026-11-18T18:35:38Z
2026-11-18T13:37:35-05:00:   check qemu-server.conf.blob
2026-11-18T13:37:35-05:00:   check drive-scsi0.img.fidx
2026-11-18T13:48:27-05:00:   verified 57346.65/114060.00 MiB in 651.48 seconds, speed 88.03/175.08 MiB/s (3 errors)
2026-11-18T13:48:27-05:00: verify External:vm/102/2026-11-18T18:35:38Z/drive-scsi0.img.fidx failed: chunks could not be verified
2026-11-18T13:48:27-05:00: Failed to verify the following snapshots/groups:
2026-11-18T13:48:27-05:00:     vm/102/2026-11-18T18:35:38Z
2026-11-18T13:48:27-05:00: TASK ERROR: verification failed - please check the log for details
 
Last edited:
I'd start with the verify task on the PBS host that owns External. The excerpt narrows it down to chunks referenced by drive-scsi0.img.fidx. In the full task log, look between that archive's check line and the summary with the error count. The detail there can distinguish a failed chunk read from a check that failed after the chunk was read. Could you share the first detailed error and a few lines around it, if present? PBS verify code

New snapshots in the same datastore can refer to chunks already stored there, so seeing the same problem after a fresh backup can still point to the datastore. If the log identifies the chunks, compare the IDs across failed snapshots. If it reports an I/O error, check the PBS host's storage logs at that timestamp. Those details would help narrow it down before changing the VM disk again. PBS technical overview
 
Hi,
please do also provide the output of proxmox-backup-manager version --verbose and details about the storage backing the PBS datastore you are backing up to.

I'm trying to get PBS working smoothly but I keep running into a snag. I'm able to backup the VM/CT's fine, but it occurs often that the backup can't be verified. I've tried removing the backups and taking new ones with no change, they keep failing. One just keeps failing the initial backup verification.
Please make sure to do a full verification of the datastore before any new backup and add the full task log as attachement. If one of the last snapshot in the respective group is corrupt, but not marked as such, a subsequent backup run might reuse chunks which are corrupt already. If the verify state is failed, a new backup must re-upload any chunk though.

Further, also note that verification errors sometime arise due to bad memory, so also check your RAM by a prolonged memory test to rule that out.
 
Hi,
please do also provide the output of proxmox-backup-manager version --verbose and details about the storage backing the PBS datastore you are backing up to.


Please make sure to do a full verification of the datastore before any new backup and add the full task log as attachement. If one of the last snapshot in the respective group is corrupt, but not marked as such, a subsequent backup run might reuse chunks which are corrupt already. If the verify state is failed, a new backup must re-upload any chunk though.

Further, also note that verification errors sometime arise due to bad memory, so also check your RAM by a prolonged memory test to rule that out.
I'm using a container on my TrueNAS computer running PBS. This is backing up to a pool with its own generic dataset for this specific purpose. Permissions should be correct, and since it's only a couple vm's/ct's failing it would seem they are.

This is what I got for the version command:
Code:
root@PBS:~# proxmox-backup-manager version --verbose
proxmox-backup                     unknown       running kernel: 6.12.15-production+truenas
proxmox-backup-server              4.2.11-1      running version: 4.2.11                  
ifupdown2                          3.3.0-1+pmx14                                          
libjs-extjs                        7.0.0-9                                                
proxmox-backup-docs                4.2.11-1                                              
proxmox-backup-client              unknown                                                
proxmox-enterprise-support-keyring 1.1                                                    
proxmox-mail-forward               1.1.0                                                  
proxmox-mini-journalreader         1.7                                                    
proxmox-offline-mirror-helper      0.8.0                                                  
proxmox-widget-toolkit             5.2.13                                                
pve-xtermjs                        6.0.0-3                                                
smartmontools                      7.5-pve2                                              
zfsutils-linux                     2.4.4-pve1

I've attached the full verification log as well. At lot that were marked as verified are not longer verified... Most appear to be VM's, no other containers failed. I guess it's probably the datastore itself then? Ugh, this is so annoying. I just wanted smaller backup sizes :confused:. Any recommendations on the best way to do a memory test? I'm sure I can find a way to do it but wasn't sure if you had something specific in mind!

I might try a switch to a bare-metal device, maybe that'll be better. I didn't do that first because it would be a super jank setup but, if it works then...
 

Attachments

Last edited: