Backup job failed with err -125 - Operation canceled

[d.doianov]

Hello, everyone!
Today I've faced an unknown problem with one of mine vm's:

INFO: starting new backup job: vzdump 101 --node node1 --storage backup-C1 --notes-template '{{guestname}}' --mode stop --remove 0
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-11-10 12:06:48
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: EX56
INFO: include disk 'scsi0' 'local-m2-b:vm-20561-disk-3' 60G
INFO: include disk 'scsi1' 'local-data:vm-20561-disk-0' 350G
INFO: include disk 'scsi2' 'local-m2-b:vm-20561-disk-2' 10G
INFO: include disk 'efidisk0' 'local-m2-a:vm-20561-disk-0' 4M
INFO: include disk 'tpmstate0' 'local-m2-a:vm-20561-disk-1' 4M
INFO: creating Proxmox Backup Server archive 'vm/101/2023-11-10T10:06:48Z'
INFO: starting kvm to execute backup task
swtpm_setup: Not overwriting existing state file.
INFO: attaching TPM drive to QEMU for backup
INFO: started backup task '67b76d16-bdbe-4e84-b8c3-c2e7097c5138'
INFO: efidisk0: dirty-bitmap status: created new
INFO: scsi0: dirty-bitmap status: created new
INFO: scsi1: dirty-bitmap status: created new
INFO: scsi2: dirty-bitmap status: created new
INFO: tpmstate0-backup: dirty-bitmap status: created new
INFO:   0% (556.0 MiB of 420.0 GiB) in 3s, read: 185.3 MiB/s, write: 181.3 MiB/s
INFO:   1% (4.4 GiB of 420.0 GiB) in 26s, read: 170.1 MiB/s, write: 146.1 MiB/s
INFO:   1% (5.4 GiB of 420.0 GiB) in 31s, read: 212.0 MiB/s, write: 102.4 MiB/s
ERROR: job failed with err -125 - Operation canceled
INFO: aborting backup job
INFO: stopping kvm after backup task
trying to acquire lock...
 OK
ERROR: Backup of VM 101 failed - job failed with err -125 - Operation canceled
INFO: Failed at 2023-11-10 12:07:26
INFO: Backup job finished with errors
TASK ERROR: job errors

All vm disks seams fine - they easily can migrate between physical drives. VM - working fine, but backups failing in every mode, and I can´t find error code with google( Please advise.

 

[s.lendl]

At the time of the backup, do you see any indications in the journal why the backup was canceled?

Please upload the journal.log to find out more.

journalctl --since "2023-11-10" --until "2023-11-11" > journal.log

 

[d.doianov]

Hi, stfl.
Thanks for your reply.
I´d doublechecked journal.log and find out that there was an error from nvme drive durring the backup proces.

Nov 10 12:06:35 node1 kernel: blk_update_request: critical medium error, dev nvme1n1, sector 2813057080 op 0x0:(READ) flags 0x4000 phys_seg 123 prio class 2
Nov 10 12:06:36 node1 kernel: vmbr0: port 2(tap101i0) entered disabled state

Wearout of that drive is 10% and I've don't even think about that direction.
After moving all vm drives to another storage - all good.

Thx