Backups Failing - Clone Fails too

[tbh]

I get alerts from my system on the status of my nightly backups.

All of them except for 1 completes successfully. The one that is failing to backup gives an error at 50% completion of the backup


INFO: starting new backup job: vzdump 108 --node pve00 --remove 0 --storage local --compress zstd --mode snapshot
INFO: Starting Backup of VM 108 (qemu)
INFO: Backup started at 2021-12-28 14:06:04
INFO: status = running
INFO: VM Name: xyzVM
INFO: include disk 'scsi0' 'local-zfs:vm-108-disk-1' 32G
INFO: include disk 'efidisk0' 'local-zfs:vm-108-disk-0' 1M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-108-2021_12_28-14_06_04.vma.zst'
INFO: started backup task 'd57b3fab-32c7-4af6-8d9d-83a9c92309b2'
INFO: resuming VM again
INFO: 0% (217.5 MiB of 32.0 GiB) in 3s, read: 72.5 MiB/s, write: 58.0 MiB/s
INFO: 1% (495.3 MiB of 32.0 GiB) in 6s, read: 92.6 MiB/s, write: 65.4 MiB/s
...
...
INFO: 48% (15.4 GiB of 32.0 GiB) in 12m 56s, read: 21.7 MiB/s, write: 17.1 MiB/s
INFO: 49% (15.7 GiB of 32.0 GiB) in 13m 24s, read: 11.0 MiB/s, write: 7.9 MiB/s
INFO: 50% (16.0 GiB of 32.0 GiB) in 13m 54s, read: 11.4 MiB/s, write: 9.3 MiB/s
INFO: 50% (16.2 GiB of 32.0 GiB) in 14m 10s, read: 11.4 MiB/s, write: 8.0 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 108 failed - job failed with err -5 - Input/output error

A clone of the VM fails like this with the default settings:


create full clone of drive efidisk0 (local-zfs:vm-108-disk-0)
drive mirror is starting for drive-efidisk0
drive-efidisk0: Cancelling block job
drive-efidisk0: Done.
TASK ERROR: clone failed: block job (mirror) error: drive-efidisk0: 'mirror' has been cancelled

Attempting to clone to an external nfs disk yeids the following results:

create full clone of drive efidisk0 (local-zfs:vm-108-disk-0)
Formatting '/mnt/pve/nfss/images/113/vm-113-disk-0.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=131072 lazy_refcounts=off refcount_bits=16
drive mirror is starting for drive-efidisk0
drive-efidisk0: transferred 128.0 KiB of 128.0 KiB (100.00%) in 0s
drive-efidisk0: transferred 128.0 KiB of 128.0 KiB (100.00%) in 1s, ready
all 'mirror' jobs are ready
create full clone of drive scsi0 (local-zfs:vm-108-disk-1)
Formatting '/mnt/pve/nfss/images/113/vm-113-disk-1.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off preallocation=metadata compression_type=zlib size=34359738368 lazy_refcounts=off refcount_bits=16
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 0.0 B of 32.0 GiB (0.00%) in 0s
drive-scsi0: transferred 49.0 MiB of 32.0 GiB (0.15%) in 1s
drive-scsi0: transferred 86.0 MiB of 32.0 GiB (0.26%) in 2s
...
...
drive-scsi0: transferred 16.0 GiB of 32.0 GiB (49.96%) in 7m 6s
drive-scsi0: transferred 16.0 GiB of 32.0 GiB (50.06%) in 7m 7s
drive-scsi0: transferred 16.1 GiB of 32.0 GiB (50.22%) in 7m 8s
drive-efidisk0: Cancelling block job
drive-scsi0: Cancelling block job
drive-efidisk0: Done.
drive-scsi0: Done.
TASK ERROR: clone failed: block job (mirror) error: drive-scsi0: 'mirror' has been cancelled

Any help to get this VM to backup would be awesome.. Thanks!

 

[Kilaketia]

Hi, BTRFS on hw raid too ? I've got the same error.
To fix the backup, create a new drive, clone the first one to the second, modify fstab, delete the first one, boot on the new one. And it will backup.

I've noticed that I've got checksum errors on a disk of a vm too but it came backup after two days, don't really know why.
You can check with a btrfs scrub the fs and in dmesg you will see which files are corrupted.

I'm on proxmox 7.1-7 on a DL380 Gen10.