Backup Job failed with -err 61

[dirkdiggler]

I have an old neglected node running 6.4-15. I am trying to migrate the two remaining VM guests to a new host running 7.3-6. The plan was to backup the VMs and then load them up in the new environment. However, backup is consistently failing.

INFO: starting new backup job: vzdump 101 --mode stop --storage pve02-nfs --node pve --compress zstd --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-03-23 17:01:36
INFO: status = running
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: haosova-8.0.rc2
INFO: include disk 'sata0' 'local-lvm:vm-101-disk-1' 32G
INFO: include disk 'efidisk0' 'local-lvm:vm-101-disk-0' 4M
INFO: stopping vm
INFO: creating vzdump archive '/mnt/pve/pve02-nfs/dump/vzdump-qemu-101-2023_03_23-17_01_36.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '5a9f9458-5d53-4849-b69e-6a74575e94bc'
INFO: resuming VM again after 18 seconds
INFO:   1% (411.4 MiB of 32.0 GiB) in 3s, read: 137.1 MiB/s, write: 112.8 MiB/s
INFO:   2% (868.9 MiB of 32.0 GiB) in 6s, read: 152.5 MiB/s, write: 96.8 MiB/s
INFO:   3% (1.1 GiB of 32.0 GiB) in 9s, read: 99.2 MiB/s, write: 98.9 MiB/s
INFO:   4% (1.4 GiB of 32.0 GiB) in 12s, read: 89.0 MiB/s, write: 88.7 MiB/s
INFO:   5% (1.7 GiB of 32.0 GiB) in 15s, read: 93.7 MiB/s, write: 93.5 MiB/s
INFO:   6% (2.0 GiB of 32.0 GiB) in 19s, read: 77.6 MiB/s, write: 74.3 MiB/s
INFO:   7% (2.3 GiB of 32.0 GiB) in 23s, read: 80.6 MiB/s, write: 80.6 MiB/s
INFO:   8% (2.6 GiB of 32.0 GiB) in 27s, read: 82.5 MiB/s, write: 81.4 MiB/s
INFO:   9% (3.0 GiB of 32.0 GiB) in 30s, read: 120.3 MiB/s, write: 87.1 MiB/s
INFO:  10% (3.3 GiB of 32.0 GiB) in 33s, read: 113.3 MiB/s, write: 107.7 MiB/s
INFO:  11% (3.8 GiB of 32.0 GiB) in 36s, read: 178.7 MiB/s, write: 99.4 MiB/s
INFO:  12% (4.1 GiB of 32.0 GiB) in 39s, read: 88.3 MiB/s, write: 73.1 MiB/s
INFO:  13% (4.3 GiB of 32.0 GiB) in 42s, read: 86.8 MiB/s, write: 73.5 MiB/s
INFO:  14% (4.6 GiB of 32.0 GiB) in 45s, read: 102.1 MiB/s, write: 77.8 MiB/s
INFO:  15% (5.0 GiB of 32.0 GiB) in 48s, read: 127.7 MiB/s, write: 88.4 MiB/s
INFO:  16% (5.3 GiB of 32.0 GiB) in 51s, read: 90.2 MiB/s, write: 89.9 MiB/s
INFO:  17% (5.8 GiB of 32.0 GiB) in 54s, read: 164.5 MiB/s, write: 85.7 MiB/s
INFO:  18% (5.9 GiB of 32.0 GiB) in 56s, read: 66.5 MiB/s, write: 66.3 MiB/s
ERROR: job failed with err -61 - No data available
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - job failed with err -61 - No data available
INFO: Failed at 2023-03-23 17:02:53
INFO: Backup job finished with errors
TASK ERROR: job errors

This particular VM fails at 18% every time. The other VM fails at 30%. I have tried backing up to local as well as NFS, so I don't think it's the NFS server. I also rebooted the NFS server for good measure but still no success.

I have checked the underlying disk with a smartctl short test which returned Completed without error

Any suggestions for troubleshooting next steps? Or should I abandon this path, try to upgrade the environment to PVE7 and then migrate the guests directly?

 

[KathouQC]

I am in PVE7 and I got the same issue

 

[mokaz]

Same issue here on a couple of VMs, actually previously "restored" VMs -- tried local, PBS, CIFS targetd backups / no luck -- this on PVE 7.4-3:

INFO: 28% (56.7 GiB of 201.0 GiB) in 6m 18s, read: 94.2 MiB/s, write: 93.6 MiB/s
ERROR: job failed with err -61 - No data available
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM XXX failed - job failed with err -61 - No data available
INFO: Failed at 2023-04-02 06:54:05
INFO: Backup job finished with errors
TASK ERROR: job errors

Have you found a way out ?
thanks

 

[Choppel]

This happened for me a couple of times now on different PVE instances. The only way of getting the vm to work again was to create a new virtual disk, backing up the content of the old virtual disk with Clonezilla into the new disk and then removing the old disk.

 

[Choppel]

PS: There is most likely a problem with the physical disk. Have a look at the syslog during backup.

 

[mokaz]

Hey Choppel,
Thanks for the update, actually that's the last VM giving me issues.. I'm currently in --rescue mode on clonezilla and can confirm that supposedly there are some lost sectors on the LV.. I'll see if I can rescue that and if not will resetup the box. I'm not convinced that these bad sectors are really happening at the physical layer though. Let's see. Thanks !

 

[mokaz]

Indeed, out of that through clonezilla, will investigate the NVMe physical side as well though:

INFO: 100% (201.0 GiB of 201.0 GiB) in 10m 59s, read: 228.4 MiB/s, write: 125.3 MiB/s
INFO: backup is sparse: 108.28 GiB (53%) total zero data
INFO: transferred 201.00 GiB in 659 seconds (312.3 MiB/s)
INFO: archive file size: 48.98GB
INFO: adding notes to backup
INFO: Finished Backup of VM 000 (00:11:00)
INFO: Backup finished at 2023-04-14 18:45:55
INFO: Backup job finished successfully
TASK OK

Thanks,
Cheers,
M.