Backup job failed with err -125 - Operation canceled

[pain]

Hi Everyone,

I am in the process of migrating the VMs from one proxmox server to another so I can update the proxmox to 8.1.
I am creating the backups of VM, then send the backup to the other proxmox server and then restore that VM. The virtual machines are running critical micro-services due to which I can't afford the downtime for more than 5-10 minutes. Also, those machines are very hard to set up from the scratch which is itself a full time job.

When I create a backup I get then below logs.

2024-01-29 12:52:39 INFO: Starting Backup of VM 198 (qemu) 2024-01-29 12:52:39 INFO: status = stopped 2024-01-29 12:52:39 INFO: backup mode: stop 2024-01-29 12:52:39 INFO: ionice priority: 7 2024-01-29 12:52:39 INFO: VM Name: hscn-win10-vm-e-rs-dorota-live 2024-01-29 12:52:39 INFO: include disk 'scsi0' 'local-lvm:vm-198-disk-0' 60G 2024-01-29 12:52:39 INFO: include disk 'tpmstate0' 'local-lvm:vm-198-disk-1' 4M 2024-01-29 12:52:39 INFO: snapshots found (not included into backup) 2024-01-29 12:52:39 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-198-2024_01_29-12_52_39.vma.gz' 2024-01-29 12:52:39 INFO: starting kvm to execute backup task 2024-01-29 12:52:40 INFO: attaching TPM drive to QEMU for backup 2024-01-29 12:52:40 INFO: started backup task '66435448-a76c-4adc-8e11-7cdda043473a' 2024-01-29 12:52:43 INFO: 0% (177.6 MiB of 60.0 GiB) in 3s, read: 59.2 MiB/s, write: 44.3 MiB/s 2024-01-29 12:52:49 INFO: 0% (432.0 MiB of 60.0 GiB) in 9s, read: 42.4 MiB/s, write: 40.2 MiB/s 2024-01-29 12:52:49 ERROR: job failed with err -125 - Operation canceled 2024-01-29 12:52:49 INFO: aborting backup job 2024-01-29 12:52:49 INFO: stopping kvm after backup task 2024-01-29 12:52:50 ERROR: Backup of VM 198 failed - job failed with err -125 - Operation canceled


The qm config is below

agent: 1 boot: order=scsi0;net0;ide0 cores: 8 machine: pc-q35-6.1 memory: 8192 meta: creation-qemu=6.1.0,ctime=1662558585 name: hscn-win10-vm-e-rs-dorota-live net0: e1000=42:31:43:41:17:0F,bridge=vmbr0,firewall=1 numa: 0 ostype: win11 parent: Before_Update scsi0: local-lvm:vm-198-disk-0,cache=writeback,discard=on,size=60G scsihw: virtio-scsi-pci smbios1: uuid=c8eafa2c-aac5-43a9-9c49-b968af649898 sockets: 1 tpmstate0: local-lvm:vm-198-disk-1,size=4M,version=v2.0 usb0: host=1-5,usb3=1 vmgenid: 963fef7d-aca5-4148-890f-f3f7c93bc831

I am at the mercy of smart and sharp people in this forum.Please help me to get rid of the above error.

 

[fiona]

Hi,
the obvious guess is that there is an issue with the underlying storage, either where the VM images are or backup target. Check your system logs/journal for any errors and also check the health of your physical disks, e.g. with smartctl. While the VM is down, you can also try reading your VM image from start to finish with qemu-img convert (which also has a --salvage option).