Backup error - Job failed with err 125

[ExgaS]

Hi everyone
I have problems making a backup, the jobs end with error 125.
Here is the log:

INFO: starting new backup job: vzdump 200 --storage local --mode snapshot --compress zstd --remove 0 --node ****
INFO: Starting Backup of VM 200 (qemu)
INFO: Backup started at 2021-09-19 13:18:58
INFO: status = running
INFO: VM Name: ****
INFO: include disk 'ide1' '******:vm-200-disk-0' 430G
INFO: exclude disk 'ide3' '********:vm-200-disk-0' (backup=no)
INFO: include disk 'virtio0' '*******:vm-200-disk-0' 55G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-200-2021_09_19-13_18_58.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'bf85509a-da56-4280-aa21-5688763b2fd3'
INFO: resuming VM again
INFO:   2% (13.3 GiB of 485.0 GiB) in 3s, read: 4.4 GiB/s, write: 21.5 MiB/s
INFO:   3% (17.5 GiB of 485.0 GiB) in 6s, read: 1.4 GiB/s, write: 0 B/s
INFO:   4% (21.7 GiB of 485.0 GiB) in 9s, read: 1.4 GiB/s, write: 0 B/s
INFO:   5% (27.3 GiB of 485.0 GiB) in 12s, read: 1.9 GiB/s, write: 0 B/s
INFO:   6% (31.7 GiB of 485.0 GiB) in 15s, read: 1.5 GiB/s, write: 0 B/s
INFO:   7% (37.9 GiB of 485.0 GiB) in 19s, read: 1.5 GiB/s, write: 0 B/s
INFO:   8% (42.6 GiB of 485.0 GiB) in 22s, read: 1.6 GiB/s, write: 0 B/s
INFO:  10% (49.8 GiB of 485.0 GiB) in 26s, read: 1.8 GiB/s, write: 0 B/s
INFO:  11% (53.9 GiB of 485.0 GiB) in 29s, read: 1.4 GiB/s, write: 0 B/s
INFO:  12% (59.2 GiB of 485.0 GiB) in 33s, read: 1.3 GiB/s, write: 70.2 MiB/s
INFO:  13% (63.2 GiB of 485.0 GiB) in 36s, read: 1.3 GiB/s, write: 57.3 KiB/s
INFO:  14% (69.0 GiB of 485.0 GiB) in 40s, read: 1.5 GiB/s, write: 8.0 KiB/s
INFO:  15% (73.4 GiB of 485.0 GiB) in 43s, read: 1.4 GiB/s, write: 0 B/s
INFO:  16% (79.5 GiB of 485.0 GiB) in 47s, read: 1.5 GiB/s, write: 131.0 KiB/s
INFO:  17% (85.5 GiB of 485.0 GiB) in 51s, read: 1.5 GiB/s, write: 144.0 KiB/s
INFO:  18% (89.7 GiB of 485.0 GiB) in 54s, read: 1.4 GiB/s, write: 21.3 KiB/s
INFO:  19% (95.9 GiB of 485.0 GiB) in 57s, read: 2.1 GiB/s, write: 53.3 KiB/s
INFO:  20% (99.8 GiB of 485.0 GiB) in 1m, read: 1.3 GiB/s, write: 106.7 KiB/s
INFO:  21% (105.1 GiB of 485.0 GiB) in 1m 4s, read: 1.3 GiB/s, write: 112.0 KiB/s
INFO:  22% (111.1 GiB of 485.0 GiB) in 1m 7s, read: 2.0 GiB/s, write: 229.3 KiB/s
INFO:  23% (115.2 GiB of 485.0 GiB) in 1m 10s, read: 1.4 GiB/s, write: 64.0 KiB/s
INFO:  24% (120.0 GiB of 485.0 GiB) in 1m 13s, read: 1.6 GiB/s, write: 50.7 KiB/s
INFO:  25% (123.8 GiB of 485.0 GiB) in 1m 16s, read: 1.3 GiB/s, write: 64.0 KiB/s
INFO:  26% (129.7 GiB of 485.0 GiB) in 1m 20s, read: 1.5 GiB/s, write: 94.0 KiB/s
INFO:  27% (134.1 GiB of 485.0 GiB) in 1m 23s, read: 1.5 GiB/s, write: 64.0 KiB/s
INFO:  28% (140.0 GiB of 485.0 GiB) in 1m 27s, read: 1.5 GiB/s, write: 62.0 KiB/s
INFO:  30% (145.9 GiB of 485.0 GiB) in 1m 31s, read: 1.5 GiB/s, write: 88.0 KiB/s
INFO:  31% (150.6 GiB of 485.0 GiB) in 1m 34s, read: 1.6 GiB/s, write: 1.2 MiB/s
INFO:  32% (156.5 GiB of 485.0 GiB) in 1m 38s, read: 1.5 GiB/s, write: 60.0 KiB/s
INFO:  33% (163.2 GiB of 485.0 GiB) in 1m 42s, read: 1.7 GiB/s, write: 103.0 KiB/s
INFO:  34% (167.0 GiB of 485.0 GiB) in 1m 45s, read: 1.3 GiB/s, write: 45.3 KiB/s
INFO:  35% (172.9 GiB of 485.0 GiB) in 1m 49s, read: 1.5 GiB/s, write: 95.0 KiB/s
INFO:  36% (177.6 GiB of 485.0 GiB) in 1m 52s, read: 1.6 GiB/s, write: 40.0 KiB/s
INFO:  37% (183.6 GiB of 485.0 GiB) in 1m 56s, read: 1.5 GiB/s, write: 76.0 KiB/s
INFO:  39% (189.3 GiB of 485.0 GiB) in 2m, read: 1.4 GiB/s, write: 57.0 KiB/s
INFO:  40% (195.5 GiB of 485.0 GiB) in 2m 4s, read: 1.5 GiB/s, write: 32.0 KiB/s
INFO:  41% (201.4 GiB of 485.0 GiB) in 2m 8s, read: 1.5 GiB/s, write: 0 B/s
INFO:  42% (208.1 GiB of 485.0 GiB) in 2m 12s, read: 1.7 GiB/s, write: 70.0 KiB/s
INFO:  43% (212.2 GiB of 485.0 GiB) in 2m 15s, read: 1.3 GiB/s, write: 0 B/s
INFO:  44% (216.0 GiB of 485.0 GiB) in 2m 18s, read: 1.3 GiB/s, write: 0 B/s
INFO:  45% (222.8 GiB of 485.0 GiB) in 2m 22s, read: 1.7 GiB/s, write: 0 B/s
INFO:  47% (228.7 GiB of 485.0 GiB) in 2m 26s, read: 1.5 GiB/s, write: 0 B/s
INFO:  48% (234.7 GiB of 485.0 GiB) in 2m 30s, read: 1.5 GiB/s, write: 16.0 KiB/s
INFO:  49% (240.7 GiB of 485.0 GiB) in 2m 34s, read: 1.5 GiB/s, write: 16.0 KiB/s
INFO:  50% (242.6 GiB of 485.0 GiB) in 2m 38s, read: 492.1 MiB/s, write: 120.2 MiB/s
INFO:  51% (247.5 GiB of 485.0 GiB) in 3m 10s, read: 154.5 MiB/s, write: 149.0 MiB/s
INFO:  52% (252.3 GiB of 485.0 GiB) in 3m 44s, read: 146.0 MiB/s, write: 141.3 MiB/s
INFO:  53% (257.1 GiB of 485.0 GiB) in 4m 33s, read: 99.7 MiB/s, write: 92.6 MiB/s
INFO:  54% (262.0 GiB of 485.0 GiB) in 6m 13s, read: 50.6 MiB/s, write: 43.4 MiB/s
INFO:  55% (266.8 GiB of 485.0 GiB) in 7m 22s, read: 71.3 MiB/s, write: 69.2 MiB/s
INFO:  56% (272.3 GiB of 485.0 GiB) in 7m 44s, read: 254.4 MiB/s, write: 135.3 MiB/s
INFO:  57% (276.5 GiB of 485.0 GiB) in 9m 39s, read: 37.1 MiB/s, write: 35.8 MiB/s
INFO:  58% (281.4 GiB of 485.0 GiB) in 12m 24s, read: 30.9 MiB/s, write: 29.8 MiB/s
INFO:  59% (286.2 GiB of 485.0 GiB) in 15m 19s, read: 27.9 MiB/s, write: 27.1 MiB/s
INFO:  60% (291.1 GiB of 485.0 GiB) in 16m 43s, read: 60.2 MiB/s, write: 59.0 MiB/s
INFO:  61% (296.0 GiB of 485.0 GiB) in 18m 19s, read: 51.8 MiB/s, write: 50.0 MiB/s
INFO:  62% (300.9 GiB of 485.0 GiB) in 20m 47s, read: 33.8 MiB/s, write: 32.0 MiB/s
INFO:  63% (305.6 GiB of 485.0 GiB) in 22m 5s, read: 62.2 MiB/s, write: 61.2 MiB/s
INFO:  64% (310.4 GiB of 485.0 GiB) in 24m 13s, read: 38.4 MiB/s, write: 37.7 MiB/s
INFO:  65% (315.4 GiB of 485.0 GiB) in 25m 52s, read: 51.2 MiB/s, write: 49.6 MiB/s
INFO:  66% (320.2 GiB of 485.0 GiB) in 28m 15s, read: 34.6 MiB/s, write: 33.9 MiB/s
INFO:  67% (325.0 GiB of 485.0 GiB) in 30m 22s, read: 38.4 MiB/s, write: 37.1 MiB/s
INFO:  68% (329.8 GiB of 485.0 GiB) in 31m 22s, read: 82.8 MiB/s, write: 82.1 MiB/s
INFO:  69% (334.8 GiB of 485.0 GiB) in 33m 32s, read: 39.4 MiB/s, write: 39.0 MiB/s
INFO:  70% (339.7 GiB of 485.0 GiB) in 35m 46s, read: 37.0 MiB/s, write: 36.1 MiB/s
INFO:  71% (344.4 GiB of 485.0 GiB) in 37m 34s, read: 45.2 MiB/s, write: 44.8 MiB/s
INFO:  72% (349.3 GiB of 485.0 GiB) in 39m 19s, read: 47.9 MiB/s, write: 46.9 MiB/s
INFO:  72% (353.3 GiB of 485.0 GiB) in 41m 5s, read: 37.7 MiB/s, write: 37.0 MiB/s
ERROR: job failed with err -125 - Operation canceled
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 200 failed - job failed with err -125 - Operation canceled
INFO: Failed at 2021-09-19 14:00:12
INFO: Backup job finished with errors
TASK ERROR: job errors

I tried to google it, but i haven't found too much info
It always fails at 72% and there is enough space where i'm saving the backup.
Does anyone see what could be the reason ? Thanks

 

[dcsapak]

this is a normal vzdump (non-pbs) backup yes? in that case, it can only be the local disk where the backup is written to, or the disk where the original disk is read from
how does the vm config look like?
maybe the source disk is not completely readable?

 

[ExgaS]

this is a normal vzdump (non-pbs) backup yes? in that case, it can only be the local disk where the backup is written to, or the disk where the original disk is read from
how does the vm config look like?
maybe the source disk is not completely readable?

Hi, yes its normal vzdump (non-pbs) that i'm doing with PVE, now im noticing that is the wrong forum, since its related to PVE, sorry for that, i dont know how but i posted on the wrong forum.
I checked and the ZFS pool where is the VM, and its showing errors, so i guess the problems that i'm having with the backup are related to that ?

 

[dcsapak]

I checked and the ZFS pool where is the VM, and its showing errors, so i guess the problems that i'm having with the backup are related to that ?

yes that can be the reason