[SOLVED] Backup Error - job failed with err -125 - Operation cancel

S

sitruk

New Member
Mar 10, 2023
11
1
1
Hello,
I didn't post this in the Backup Server section since this isn't a dedicated backup server. I'm sorry if this was not the correct thing.

I keep getting this error for a Windows VM backup. ERROR: Backup of VM 103 failed - job failed with err -125 - Operation canceled

I'm having trouble finding anything about this error. I'm also not sure if there's a way to pull more error details from the console.

I'm sorry if this is all id10t stuff. I appreciate your help. Thank you.
 
Hi,
can you share the full backup log? This error can indicate an issue reading the VM's disk. Please check your underlying physical disks and for errors in the system logs. Can you check if you can read the whole images (while the VM is down) with qemu-img convert? If not, you can use the --salvage option to skip the bad parts.
 
Hello Fiona! Thank you for the reply.

It does look like some kind of disk read error since it stops before 100% during part of the backup.
Can you check if you can read the whole images (while the VM is down) with qemu-img convert? If not, you can use the --salvage option to skip the bad parts.
Click to expand...
I'm sorry, I'm kind of new to PVE. Can you elaborate on how I can use this command? It seems to require additional parameters that I'm not sure of. I'd rather be safe than mess something up by accident.

INFO: starting new backup job: vzdump 101 103 104 --compress zstd --prune-backups 'keep-last=4' --storage SynoNFS01 --notes-template '{{guestname}}' --quiet 1 --mode snapshot --mailnotification failure
INFO: skip external VMs: 104
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2023-06-11 21:00:00
INFO: status = running
INFO: VM Name: homeassistant
INFO: include disk 'scsi0' 'local-zfs:vm-101-disk-13' 32G
INFO: include disk 'efidisk0' 'local-zfs:vm-101-disk-12' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/SynoNFS01/dump/vzdump-qemu-101-2023_06_11-21_00_00.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '680f2fa1-c4a7-41e9-ba2d-a56e3ff576fa'
INFO: resuming VM again
INFO: 1% (543.8 MiB of 32.0 GiB) in 3s, read: 181.2 MiB/s, write: 147.2 MiB/s
INFO: 3% (1.2 GiB of 32.0 GiB) in 6s, read: 219.9 MiB/s, write: 152.8 MiB/s
INFO: 4% (1.6 GiB of 32.0 GiB) in 9s, read: 139.0 MiB/s, write: 135.3 MiB/s
INFO: 6% (2.0 GiB of 32.0 GiB) in 12s, read: 146.2 MiB/s, write: 134.2 MiB/s
INFO: 7% (2.4 GiB of 32.0 GiB) in 15s, read: 123.2 MiB/s, write: 107.8 MiB/s
INFO: 9% (3.1 GiB of 32.0 GiB) in 18s, read: 233.2 MiB/s, write: 139.4 MiB/s
INFO: 10% (3.5 GiB of 32.0 GiB) in 21s, read: 147.4 MiB/s, write: 134.8 MiB/s
INFO: 12% (3.9 GiB of 32.0 GiB) in 24s, read: 124.5 MiB/s, write: 118.9 MiB/s
INFO: 13% (4.3 GiB of 32.0 GiB) in 27s, read: 136.5 MiB/s, write: 130.4 MiB/s
INFO: 14% (4.7 GiB of 32.0 GiB) in 30s, read: 145.0 MiB/s, write: 130.1 MiB/s
INFO: 16% (5.2 GiB of 32.0 GiB) in 33s, read: 163.1 MiB/s, write: 136.0 MiB/s
INFO: 17% (5.7 GiB of 32.0 GiB) in 36s, read: 172.8 MiB/s, write: 132.5 MiB/s
INFO: 19% (6.4 GiB of 32.0 GiB) in 39s, read: 236.8 MiB/s, write: 114.7 MiB/s
INFO: 21% (6.9 GiB of 32.0 GiB) in 42s, read: 190.9 MiB/s, write: 154.2 MiB/s
INFO: 22% (7.4 GiB of 32.0 GiB) in 45s, read: 149.8 MiB/s, write: 134.7 MiB/s
INFO: 25% (8.1 GiB of 32.0 GiB) in 48s, read: 247.7 MiB/s, write: 140.1 MiB/s
INFO: 26% (8.5 GiB of 32.0 GiB) in 51s, read: 148.6 MiB/s, write: 138.7 MiB/s
INFO: 27% (8.9 GiB of 32.0 GiB) in 54s, read: 143.8 MiB/s, write: 132.2 MiB/s
INFO: 29% (9.4 GiB of 32.0 GiB) in 57s, read: 157.1 MiB/s, write: 138.6 MiB/s
INFO: 31% (9.9 GiB of 32.0 GiB) in 1m, read: 184.9 MiB/s, write: 135.5 MiB/s
INFO: 32% (10.5 GiB of 32.0 GiB) in 1m 3s, read: 193.3 MiB/s, write: 136.6 MiB/s
INFO: 34% (11.1 GiB of 32.0 GiB) in 1m 6s, read: 190.9 MiB/s, write: 145.6 MiB/s
INFO: 35% (11.4 GiB of 32.0 GiB) in 1m 9s, read: 117.2 MiB/s, write: 115.8 MiB/s
INFO: 37% (11.9 GiB of 32.0 GiB) in 1m 12s, read: 171.6 MiB/s, write: 132.2 MiB/s
INFO: 38% (12.3 GiB of 32.0 GiB) in 1m 15s, read: 132.9 MiB/s, write: 130.6 MiB/s
INFO: 39% (12.7 GiB of 32.0 GiB) in 1m 18s, read: 151.0 MiB/s, write: 133.8 MiB/s
INFO: 41% (13.2 GiB of 32.0 GiB) in 1m 21s, read: 161.9 MiB/s, write: 141.1 MiB/s
INFO: 42% (13.6 GiB of 32.0 GiB) in 1m 24s, read: 116.0 MiB/s, write: 115.4 MiB/s
INFO: 43% (13.9 GiB of 32.0 GiB) in 1m 27s, read: 129.3 MiB/s, write: 128.0 MiB/s
INFO: 45% (14.6 GiB of 32.0 GiB) in 1m 30s, read: 226.0 MiB/s, write: 112.2 MiB/s
INFO: 46% (15.0 GiB of 32.0 GiB) in 1m 33s, read: 136.5 MiB/s, write: 122.3 MiB/s
INFO: 48% (15.5 GiB of 32.0 GiB) in 1m 36s, read: 175.2 MiB/s, write: 129.1 MiB/s
INFO: 50% (16.0 GiB of 32.0 GiB) in 1m 39s, read: 178.8 MiB/s, write: 127.2 MiB/s
INFO: 51% (16.5 GiB of 32.0 GiB) in 1m 42s, read: 149.8 MiB/s, write: 146.9 MiB/s
INFO: 53% (17.1 GiB of 32.0 GiB) in 1m 45s, read: 223.5 MiB/s, write: 164.5 MiB/s
INFO: 54% (17.5 GiB of 32.0 GiB) in 1m 48s, read: 128.1 MiB/s, write: 122.7 MiB/s
INFO: 55% (17.8 GiB of 32.0 GiB) in 1m 51s, read: 118.4 MiB/s, write: 112.3 MiB/s
INFO: 57% (18.4 GiB of 32.0 GiB) in 1m 54s, read: 183.7 MiB/s, write: 163.7 MiB/s
INFO: 58% (18.7 GiB of 32.0 GiB) in 1m 57s, read: 117.2 MiB/s, write: 100.8 MiB/s
INFO: 60% (19.2 GiB of 32.0 GiB) in 2m, read: 175.2 MiB/s, write: 146.6 MiB/s
INFO: 61% (19.6 GiB of 32.0 GiB) in 2m 3s, read: 131.7 MiB/s, write: 127.1 MiB/s
INFO: 62% (20.1 GiB of 32.0 GiB) in 2m 6s, read: 154.7 MiB/s, write: 139.9 MiB/s
INFO: 64% (20.5 GiB of 32.0 GiB) in 2m 9s, read: 143.8 MiB/s, write: 138.8 MiB/s
INFO: 65% (21.0 GiB of 32.0 GiB) in 2m 12s, read: 171.6 MiB/s, write: 144.6 MiB/s
INFO: 67% (21.5 GiB of 32.0 GiB) in 2m 15s, read: 154.7 MiB/s, write: 149.3 MiB/s
INFO: 68% (21.9 GiB of 32.0 GiB) in 2m 18s, read: 137.8 MiB/s, write: 132.2 MiB/s
INFO: 69% (22.3 GiB of 32.0 GiB) in 2m 21s, read: 148.6 MiB/s, write: 136.6 MiB/s
INFO: 70% (22.7 GiB of 32.0 GiB) in 2m 24s, read: 125.7 MiB/s, write: 114.4 MiB/s
INFO: 72% (23.2 GiB of 32.0 GiB) in 2m 27s, read: 165.5 MiB/s, write: 135.0 MiB/s
INFO: 73% (23.6 GiB of 32.0 GiB) in 2m 30s, read: 146.2 MiB/s, write: 134.9 MiB/s
INFO: 75% (24.0 GiB of 32.0 GiB) in 2m 33s, read: 157.1 MiB/s, write: 150.7 MiB/s
INFO: 76% (24.4 GiB of 32.0 GiB) in 2m 36s, read: 119.6 MiB/s, write: 110.3 MiB/s
INFO: 77% (24.8 GiB of 32.0 GiB) in 2m 39s, read: 152.2 MiB/s, write: 105.4 MiB/s
INFO: 79% (25.3 GiB of 32.0 GiB) in 2m 42s, read: 160.7 MiB/s, write: 149.4 MiB/s
INFO: 80% (25.7 GiB of 32.0 GiB) in 2m 45s, read: 130.5 MiB/s, write: 126.5 MiB/s
INFO: 81% (26.1 GiB of 32.0 GiB) in 2m 48s, read: 128.1 MiB/s, write: 124.2 MiB/s
INFO: 82% (26.4 GiB of 32.0 GiB) in 2m 51s, read: 118.4 MiB/s, write: 118.4 MiB/s
INFO: 83% (26.8 GiB of 32.0 GiB) in 2m 54s, read: 143.8 MiB/s, write: 125.2 MiB/s
INFO: 85% (27.2 GiB of 32.0 GiB) in 2m 57s, read: 134.1 MiB/s, write: 133.4 MiB/s
INFO: 86% (27.6 GiB of 32.0 GiB) in 3m, read: 130.5 MiB/s, write: 130.1 MiB/s
INFO: 87% (28.0 GiB of 32.0 GiB) in 3m 3s, read: 117.2 MiB/s, write: 117.2 MiB/s
INFO: 88% (28.3 GiB of 32.0 GiB) in 3m 6s, read: 119.6 MiB/s, write: 119.6 MiB/s
INFO: 89% (28.6 GiB of 32.0 GiB) in 3m 9s, read: 100.3 MiB/s, write: 100.0 MiB/s
INFO: 90% (29.1 GiB of 32.0 GiB) in 3m 12s, read: 168.0 MiB/s, write: 131.4 MiB/s
INFO: 92% (29.5 GiB of 32.0 GiB) in 3m 15s, read: 126.9 MiB/s, write: 126.9 MiB/s
INFO: 93% (29.9 GiB of 32.0 GiB) in 3m 18s, read: 136.5 MiB/s, write: 121.1 MiB/s
INFO: 94% (30.2 GiB of 32.0 GiB) in 3m 21s, read: 114.8 MiB/s, write: 114.4 MiB/s
INFO: 95% (30.5 GiB of 32.0 GiB) in 3m 24s, read: 117.2 MiB/s, write: 116.6 MiB/s
INFO: 98% (31.4 GiB of 32.0 GiB) in 3m 27s, read: 296.0 MiB/s, write: 125.4 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 3m 30s, read: 202.5 MiB/s, write: 109.7 MiB/s
INFO: backup is sparse: 5.27 GiB (16%) total zero data
INFO: transferred 32.00 GiB in 210 seconds (156.0 MiB/s)
INFO: archive file size: 13.89GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-last=4
INFO: removing backup 'SynoNFS01:backup/vzdump-qemu-101-2023_06_08-21_00_04.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 101 (00:03:42)
INFO: Backup finished at 2023-06-11 21:03:42
INFO: Starting Backup of VM 103 (qemu)
INFO: Backup started at 2023-06-11 21:03:42
INFO: status = running
INFO: VM Name: blueiris
INFO: include disk 'sata0' 'local-zfs:vm-103-disk-1' 60G
INFO: include disk 'sata1' 'local-zfs:vm-103-disk-3' 100G
INFO: include disk 'efidisk0' 'local-zfs:vm-103-disk-0' 1M
INFO: include disk 'tpmstate0' 'local-zfs:vm-103-disk-2' 4M
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/SynoNFS01/dump/vzdump-qemu-103-2023_06_11-21_03_42.vma.zst'
INFO: attaching TPM drive to QEMU for backup
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '919347c5-d7cb-4edd-8b69-a63a02617c49'
INFO: resuming VM again
INFO: 0% (503.9 MiB of 160.0 GiB) in 3s, read: 168.0 MiB/s, write: 153.2 MiB/s
INFO: 1% (1.7 GiB of 160.0 GiB) in 14s, read: 109.7 MiB/s, write: 109.7 MiB/s
INFO: 2% (3.2 GiB of 160.0 GiB) in 28s, read: 113.9 MiB/s, write: 113.6 MiB/s
INFO: 3% (4.9 GiB of 160.0 GiB) in 44s, read: 107.2 MiB/s, write: 107.2 MiB/s
INFO: 4% (6.5 GiB of 160.0 GiB) in 59s, read: 110.2 MiB/s, write: 110.2 MiB/s
INFO: 5% (8.0 GiB of 160.0 GiB) in 1m 13s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 6% (9.6 GiB of 160.0 GiB) in 1m 28s, read: 110.9 MiB/s, write: 110.9 MiB/s
INFO: 7% (11.3 GiB of 160.0 GiB) in 1m 44s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 8% (12.9 GiB of 160.0 GiB) in 1m 59s, read: 111.2 MiB/s, write: 111.2 MiB/s
INFO: 9% (14.4 GiB of 160.0 GiB) in 2m 13s, read: 107.5 MiB/s, write: 107.5 MiB/s
INFO: 10% (16.1 GiB of 160.0 GiB) in 2m 30s, read: 98.5 MiB/s, write: 98.5 MiB/s
INFO: 11% (17.7 GiB of 160.0 GiB) in 2m 45s, read: 109.5 MiB/s, write: 109.5 MiB/s
INFO: 12% (19.2 GiB of 160.0 GiB) in 3m 1s, read: 100.4 MiB/s, write: 100.4 MiB/s
INFO: 13% (20.9 GiB of 160.0 GiB) in 3m 17s, read: 106.3 MiB/s, write: 106.3 MiB/s
INFO: 14% (22.4 GiB of 160.0 GiB) in 3m 33s, read: 97.0 MiB/s, write: 97.0 MiB/s
INFO: 15% (24.0 GiB of 160.0 GiB) in 3m 48s, read: 110.0 MiB/s, write: 110.0 MiB/s
INFO: 16% (25.7 GiB of 160.0 GiB) in 4m 4s, read: 106.0 MiB/s, write: 106.0 MiB/s
INFO: 17% (27.3 GiB of 160.0 GiB) in 4m 19s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 18% (28.9 GiB of 160.0 GiB) in 4m 34s, read: 109.2 MiB/s, write: 109.2 MiB/s
INFO: 19% (30.5 GiB of 160.0 GiB) in 4m 49s, read: 110.9 MiB/s, write: 110.9 MiB/s
INFO: 20% (32.1 GiB of 160.0 GiB) in 5m 4s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 21% (33.6 GiB of 160.0 GiB) in 5m 19s, read: 105.1 MiB/s, write: 105.1 MiB/s
INFO: 22% (35.2 GiB of 160.0 GiB) in 5m 34s, read: 110.0 MiB/s, write: 110.0 MiB/s
INFO: 23% (36.8 GiB of 160.0 GiB) in 5m 49s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 24% (38.4 GiB of 160.0 GiB) in 6m 5s, read: 102.2 MiB/s, write: 102.2 MiB/s
INFO: 25% (40.0 GiB of 160.0 GiB) in 6m 20s, read: 109.2 MiB/s, write: 109.2 MiB/s
INFO: 26% (41.7 GiB of 160.0 GiB) in 6m 36s, read: 110.3 MiB/s, write: 110.3 MiB/s
INFO: 27% (43.3 GiB of 160.0 GiB) in 6m 51s, read: 109.2 MiB/s, write: 109.2 MiB/s
INFO: 28% (44.8 GiB of 160.0 GiB) in 7m 5s, read: 109.0 MiB/s, write: 109.0 MiB/s
INFO: 29% (46.4 GiB of 160.0 GiB) in 7m 21s, read: 103.8 MiB/s, write: 103.8 MiB/s
INFO: 30% (48.1 GiB of 160.0 GiB) in 7m 37s, read: 106.3 MiB/s, write: 106.3 MiB/s
INFO: 31% (49.7 GiB of 160.0 GiB) in 7m 52s, read: 110.4 MiB/s, write: 110.4 MiB/s
INFO: 32% (51.2 GiB of 160.0 GiB) in 8m 6s, read: 110.8 MiB/s, write: 110.8 MiB/s
INFO: 33% (52.8 GiB of 160.0 GiB) in 8m 21s, read: 109.0 MiB/s, write: 109.0 MiB/s
INFO: 34% (54.4 GiB of 160.0 GiB) in 8m 36s, read: 110.0 MiB/s, write: 110.0 MiB/s
INFO: 35% (56.0 GiB of 160.0 GiB) in 8m 51s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 36% (57.7 GiB of 160.0 GiB) in 9m 7s, read: 106.0 MiB/s, write: 106.0 MiB/s
INFO: 37% (59.3 GiB of 160.0 GiB) in 9m 22s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 38% (60.9 GiB of 160.0 GiB) in 9m 37s, read: 113.1 MiB/s, write: 113.1 MiB/s
INFO: 39% (62.5 GiB of 160.0 GiB) in 9m 52s, read: 110.4 MiB/s, write: 110.4 MiB/s
INFO: 40% (64.0 GiB of 160.0 GiB) in 10m 6s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 41% (65.7 GiB of 160.0 GiB) in 10m 22s, read: 107.8 MiB/s, write: 107.8 MiB/s
INFO: 42% (67.2 GiB of 160.0 GiB) in 10m 36s, read: 111.1 MiB/s, write: 111.1 MiB/s
INFO: 43% (68.9 GiB of 160.0 GiB) in 10m 52s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 44% (70.5 GiB of 160.0 GiB) in 11m 7s, read: 108.5 MiB/s, write: 108.5 MiB/s
INFO: 45% (72.0 GiB of 160.0 GiB) in 11m 22s, read: 103.9 MiB/s, write: 103.9 MiB/s
INFO: 46% (73.7 GiB of 160.0 GiB) in 11m 38s, read: 105.4 MiB/s, write: 105.4 MiB/s
INFO: 47% (75.3 GiB of 160.0 GiB) in 11m 53s, read: 110.7 MiB/s, write: 110.7 MiB/s
INFO: 48% (76.9 GiB of 160.0 GiB) in 12m 8s, read: 110.7 MiB/s, write: 110.7 MiB/s
INFO: 49% (78.5 GiB of 160.0 GiB) in 12m 23s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO: 50% (80.0 GiB of 160.0 GiB) in 12m 38s, read: 104.9 MiB/s, write: 104.9 MiB/s
INFO: 51% (81.7 GiB of 160.0 GiB) in 12m 55s, read: 99.6 MiB/s, write: 99.6 MiB/s
INFO: 52% (83.3 GiB of 160.0 GiB) in 13m 11s, read: 101.7 MiB/s, write: 101.7 MiB/s
INFO: 53% (84.8 GiB of 160.0 GiB) in 13m 25s, read: 112.1 MiB/s, write: 112.1 MiB/s
INFO: 54% (86.5 GiB of 160.0 GiB) in 13m 41s, read: 108.1 MiB/s, write: 108.1 MiB/s
INFO: 55% (88.1 GiB of 160.0 GiB) in 13m 56s, read: 110.2 MiB/s, write: 110.2 MiB/s
INFO: 56% (89.6 GiB of 160.0 GiB) in 14m 10s, read: 111.5 MiB/s, write: 111.5 MiB/s
INFO: 57% (91.2 GiB of 160.0 GiB) in 14m 25s, read: 107.9 MiB/s, write: 107.9 MiB/s
INFO: 58% (92.9 GiB of 160.0 GiB) in 14m 43s, read: 96.5 MiB/s, write: 96.5 MiB/s
INFO: 59% (94.4 GiB of 160.0 GiB) in 14m 58s, read: 104.2 MiB/s, write: 104.2 MiB/s
INFO: 60% (96.1 GiB of 160.0 GiB) in 15m 14s, read: 105.4 MiB/s, write: 105.4 MiB/s
INFO: 61% (97.7 GiB of 160.0 GiB) in 15m 30s, read: 100.8 MiB/s, write: 100.8 MiB/s
INFO: 62% (99.3 GiB of 160.0 GiB) in 15m 45s, read: 112.9 MiB/s, write: 112.9 MiB/s
INFO: 63% (100.9 GiB of 160.0 GiB) in 16m, read: 109.5 MiB/s, write: 102.4 MiB/s
INFO: 64% (102.4 GiB of 160.0 GiB) in 16m 15s, read: 102.2 MiB/s, write: 100.3 MiB/s
INFO: 65% (104.1 GiB of 160.0 GiB) in 16m 31s, read: 105.1 MiB/s, write: 104.7 MiB/s
INFO: 66% (105.6 GiB of 160.0 GiB) in 16m 47s, read: 101.7 MiB/s, write: 99.4 MiB/s
INFO: 67% (107.2 GiB of 160.0 GiB) in 17m 3s, read: 102.9 MiB/s, write: 101.8 MiB/s
INFO: 68% (108.9 GiB of 160.0 GiB) in 17m 17s, read: 119.6 MiB/s, write: 114.3 MiB/s
INFO: 68% (110.2 GiB of 160.0 GiB) in 17m 28s, read: 119.2 MiB/s, write: 103.7 MiB/s
ERROR: job failed with err -125 - Operation canceled
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 103 failed - job failed with err -125 - Operation canceled
INFO: Failed at 2023-06-11 21:21:38
INFO: Backup job finished with errors
TASK ERROR: job errors

Jun 11 21:03:42 proxmox03 pvescheduler[1896039]: INFO: Starting Backup of VM 103 (qemu)
Jun 11 21:08:40 proxmox03 pmxcfs[1540]: [status] notice: received log
Jun 11 21:17:01 proxmox03 CRON[2070925]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 11 21:17:01 proxmox03 CRON[2070931]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jun 11 21:17:01 proxmox03 CRON[2070925]: pam_unix(cron:session): session closed for user root
Jun 11 21:21:13 proxmox03 kernel: Buffer I/O error on dev zd16, logical block 2657650, async page read
Jun 11 21:21:13 proxmox03 kernel: Buffer I/O error on dev zd16, logical block 2657650, async page read
Jun 11 21:21:13 proxmox03 zed[2169735]: eid=39142 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=79309:1:0:1328825
Jun 11 21:21:13 proxmox03 zed[2169737]: eid=39139 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=79309:1:0:1328825
Jun 11 21:21:13 proxmox03 zed[2169739]: eid=39140 class=checksum pool='rpool' vdev=nvme-eui.6479a741b05004bd-part3 algorithm=fletcher4 size=8192 offset=44937428992 priority=0 err=52 flags=0x180880 bookmark=79309:1:0:1328825
Jun 11 21:21:13 proxmox03 zed[2169740]: eid=39141 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=79309:1:0:1328825
Jun 11 21:21:38 proxmox03 pvescheduler[1896039]: ERROR: Backup of VM 103 failed - job failed with err -125 - Operation canceled
Jun 11 21:21:38 proxmox03 pvescheduler[1896039]: INFO: Backup job finished with errors
Jun 11 21:21:38 proxmox03 pvescheduler[1896039]: job errors
 
sitruk said:
Jun 11 21:03:42 proxmox03 pvescheduler[1896039]: INFO: Starting Backup of VM 103 (qemu)
Jun 11 21:08:40 proxmox03 pmxcfs[1540]: [status] notice: received log
Jun 11 21:17:01 proxmox03 CRON[2070925]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Jun 11 21:17:01 proxmox03 CRON[2070931]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Jun 11 21:17:01 proxmox03 CRON[2070925]: pam_unix(cron:session): session closed for user root
Jun 11 21:21:13 proxmox03 kernel: Buffer I/O error on dev zd16, logical block 2657650, async page read
Jun 11 21:21:13 proxmox03 kernel: Buffer I/O error on dev zd16, logical block 2657650, async page read
Jun 11 21:21:13 proxmox03 zed[2169735]: eid=39142 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=79309:1:0:1328825
Jun 11 21:21:13 proxmox03 zed[2169737]: eid=39139 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=79309:1:0:1328825
Jun 11 21:21:13 proxmox03 zed[2169739]: eid=39140 class=checksum pool='rpool' vdev=nvme-eui.6479a741b05004bd-part3 algorithm=fletcher4 size=8192 offset=44937428992 priority=0 err=52 flags=0x180880 bookmark=79309:1:0:1328825
Jun 11 21:21:13 proxmox03 zed[2169740]: eid=39141 class=data pool='rpool' priority=0 err=52 flags=0x8881 bookmark=79309:1:0:1328825
Jun 11 21:21:38 proxmox03 pvescheduler[1896039]: ERROR: Backup of VM 103 failed - job failed with err -125 - Operation canceled
Jun 11 21:21:38 proxmox03 pvescheduler[1896039]: INFO: Backup job finished with errors
Jun 11 21:21:38 proxmox03 pvescheduler[1896039]: job errors
Click to expand...
There do seem to be errors reading indeed. What does zpool status -v show? Please check the health of the underlying disks with e.g. smartctl -a /dev/XYZ (the path is what you see in the status command).

While the VM is shut down, you can use pvesm path local-zfs:vm-103-disk-1 to get the file system path and then qemu-img convert -p /dev/zvol/path/from/previous/command /path/to/target/image.raw and similar for the other disks. At least one of them should throw an error and then you can add the --salvage option to the command.
 
fiona said:
There do seem to be errors reading indeed. What does zpool status -v show? Please check the health of the underlying disks with e.g. smartctl -a /dev/XYZ (the path is what you see in the status command).

While the VM is shut down, you can use pvesm path local-zfs:vm-103-disk-1 to get the file system path and then qemu-img convert -p /dev/zvol/path/from/previous/command /path/to/target/image.raw and similar for the other disks. At least one of them should throw an error and then you can add the --salvage option to the command.
Click to expand...

Thank you! I was not understanding the commands the first time. Much more simple than I made it.

I ran those commands and it failed the salvage process bc it thinks there's no disk space. I was going to try again but to a different storage. I just don't understand why it thinks there's no space left.


root@proxmox03:~# zpool status -v
pool: rpool
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: scrub repaired 0B in 00:03:19 with 2 errors on Sun Jun 11 00:27:20 2023
config:

NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
nvme-eui.6479a741b05004bd-part3 ONLINE 0 0 14.0K

errors: Permanent errors have been detected in the following files:

rpool/data/vm-103-disk-1:<0x1>

root@proxmox03:~# pvesm path local-zfs:vm-103-disk-1
/dev/zvol/rpool/data/vm-103-disk-1

root@proxmox03:~# qemu-img convert -p /dev/zvol/rpool/data/vm-103-disk-1 /dev/zvol/rpool/data/image.raw --salvage
qemu-img: warning: error while reading offset 10885734400: Input/output error
qemu-img: warning: error while reading offset 10885734912: Input/output error
qemu-img: warning: error while reading offset 10885735424: Input/output error
qemu-img: warning: error while reading offset 10885735936: Input/output error
qemu-img: warning: error while reading offset 10885736448: Input/output error
qemu-img: warning: error while reading offset 10885736960: Input/output error
qemu-img: warning: error while reading offset 10885737472: Input/output error
qemu-img: warning: error while reading offset 10885737984: Input/output error
qemu-img: warning: error while reading offset 10885738496: Input/output error
qemu-img: warning: error while reading offset 10885739008: Input/output error
qemu-img: warning: error while reading offset 10885739520: Input/output error
qemu-img: warning: error while reading offset 10885740032: Input/output error
qemu-img: warning: error while reading offset 10885740544: Input/output error
qemu-img: warning: error while reading offset 10885741056: Input/output error
qemu-img: warning: error while reading offset 10885741568: Input/output error
qemu-img: warning: error while reading offset 10885742080: Input/output error
qemu-img: warning: error while reading offset 11927994368: Input/output error
qemu-img: warning: error while reading offset 11927994880: Input/output error
qemu-img: warning: error while reading offset 11927995392: Input/output error
qemu-img: warning: error while reading offset 11927995904: Input/output error
qemu-img: warning: error while reading offset 11927996416: Input/output error
qemu-img: warning: error while reading offset 11927996928: Input/output error
qemu-img: warning: error while reading offset 11927997440: Input/output error
qemu-img: warning: error while reading offset 11927997952: Input/output error
qemu-img: warning: error while reading offset 11927998464: Input/output error
qemu-img: warning: error while reading offset 11927998976: Input/output error
qemu-img: warning: error while reading offset 11927999488: Input/output error
qemu-img: warning: error while reading offset 11928000000: Input/output error
qemu-img: warning: error while reading offset 11928000512: Input/output error
qemu-img: warning: error while reading offset 11928001024: Input/output error
qemu-img: warning: error while reading offset 11928001536: Input/output error
qemu-img: warning: error while reading offset 11928002048: Input/output error
qemu-img: error while writing at byte 17104368128: No space left on device

root@proxmox03:~# zfs list rpool
NAME USED AVAIL REFER MOUNTPOINT
rpool 157G 304G 96K /rpool
 
sitruk said:
Code: 
root@proxmox03:~# qemu-img convert -p /dev/zvol/rpool/data/vm-103-disk-1 /dev/zvol/rpool/data/image.raw --salvage
Click to expand...
I wouldn't pick the target on the same storage (it's already got read errors, so who knows how long it will still be operational), but rather use a different one.
 
fiona said:
I wouldn't pick the target on the same storage (it's already got read errors, so who knows how long it will still be operational), but rather use a different one.
Click to expand...
Yea, I thought that too. I was functioning on wishful thinking apparently.
 
I hooked up an external SSD and was able to perform the salvage convert to the external.

I do have another VM on this host and it's not having any issues. The drive itself seems good with the exception of this one backup issue. I'm wondering if I messed up the disk image because I was having issues with the disk becoming full and hanging the VM.

Is there a way to point the existing VM disk to this location? Or should I delete the original disk and convert back to the local storage?
 
Yea, the VM won't power back on now. :(

I'm hoping I can use the salvaged image to recover since the backups were failing. Is there a process I can follow to do that? I'm struggling to find a clear way to use the disk image on the external drive.
 
sitruk said:
Yea, the VM won't power back on now. :(

I'm hoping I can use the salvaged image to recover since the backups were failing. Is there a process I can follow to do that? I'm struggling to find a clear way to use the disk image on the external drive.
Click to expand...
You can add the drive as a directory storage to Proxmox VE. Then either
  • Manually move the image to /path/to/ssd/dir/storage/images/103/vm-103-disk-9.raw (where /path/to/ssd/dir/storage/ depends on where you mounted it and you will need to create the directory with the VM ID first) and then do a qm rescan --vmid 103. This will add the disk as an unused disk to the VM configuration and you can attach it by editing it in the UI.
  • Import the drive as a new disk for the VM (but this creates an additional copy): https://pve.proxmox.com/pve-docs/chapter-qm.html#_adding_an_external_disk_image_to_a_virtual_machine
After this, change the boot order to use the new image.
 
Thank you for your assistance.

  1. I had the external set up with a ZFS pool.This wasn't working so I destroyed it and removed from the cluster.
  2. I configured a directory on the external drive as described.
  3. I ran the salvage command again, this time using the directory storage as the destination.
  4. Ran the rescan for the VM and it loaded the unused RAW disk image
  5. I enabled the "new" disk in the VM GUI, made it first boot, and disabled the old damaged disk image.
I can now boot up my VM again.
 
Glad to hear that! Please mark the thread as [SOLVED] by editing the thread and selecting the appropriate prefix. And I'd keep a close eye on the storage with the read errors!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom