ERROR: job failed with err -5 - Input/output error

[CristianG]

Hi Guys,

Proxmox - 6.3.3

Error :

INFO: starting new backup job: vzdump 107 --compress zstd --remove 0 --node node1 --mode snapshot --storage Synology
INFO: Starting Backup of VM 107 (qemu)
INFO: Backup started at 2021-02-16 19:25:00
INFO: status = running
INFO: VM Name: DC01
INFO: include disk 'ide0' 'local-lvm:vm-107-disk-0' 150G
/dev/sdb: open failed: No medium found
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/Synology/Proxmox/dump/vzdump-qemu-107-2021_02_16-19_25_00.vma.zst'
INFO: started backup task '0892f01b-6cc4-40a3-94f8-f4ba94221df4'
INFO: resuming VM again
INFO: 0% (482.1 MiB of 150.0 GiB) in 3s, read: 160.7 MiB/s, write: 110.8 MiB/s
INFO: 1% (1.6 GiB of 150.0 GiB) in 12s, read: 123.4 MiB/s, write: 112.3 MiB/s
INFO: 2% (3.1 GiB of 150.0 GiB) in 26s, read: 111.0 MiB/s, write: 110.2 MiB/s
INFO: 3% (4.5 GiB of 150.0 GiB) in 38s, read: 121.9 MiB/s, write: 117.0 MiB/s
INFO: 4% (6.1 GiB of 150.0 GiB) in 51s, read: 122.5 MiB/s, write: 121.6 MiB/s
INFO: 5% (7.5 GiB of 150.0 GiB) in 1m 3s, read: 124.1 MiB/s, write: 121.9 MiB/s
INFO: 6% (9.1 GiB of 150.0 GiB) in 1m 16s, read: 121.9 MiB/s, write: 121.0 MiB/s
INFO: 7% (10.6 GiB of 150.0 GiB) in 1m 29s, read: 122.0 MiB/s, write: 121.1 MiB/s
INFO: 8% (12.0 GiB of 150.0 GiB) in 1m 41s, read: 120.0 MiB/s, write: 118.3 MiB/s
INFO: 8% (12.8 GiB of 150.0 GiB) in 2m 21s, read: 20.6 MiB/s, write: 20.5 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
ERROR: Backup of VM 107 failed - job failed with err -5 - Input/output error
INFO: Failed at 2021-02-16 19:27:23
INFO: Backup job finished with errors
TASK ERROR: job errors


My Nas space :

Enabled Yes
Active Yes
Content VZDump backup file
Type Directory
13.89% (1.24 TiB of 8.91 TiB)

I done all my VM backup only VM 107 ouput "failed - job failed with err -5 - Input/output error"

Thank you

 

[oguz]

hi,

I done all my VM backup only VM 107 ouput "failed - job failed with err -5 - Input/output error"

if all other VMs can be backuped but not this one, then it might be an issue with the VM disk.

could you test if the backup works when you use a different backup mode (like stop mode)

 

[CristianG]

Hi,
Thank you for your response

INFO: starting new backup job: vzdump 107 --compress zstd --remove 0 --node node1 --mode stop --storage Synology
INFO: Starting Backup of VM 107 (qemu)
INFO: Backup started at 2021-02-17 15:53:35
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: DC01
INFO: include disk 'ide0' 'local-lvm:vm-107-disk-0' 150G
INFO: creating vzdump archive '/mnt/Synology/Proxmox/dump/vzdump-qemu-107-2021_02_17-15_53_35.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '8acff5a0-cc3f-4cf2-942b-154b65872473'
INFO: 0% (524.8 MiB of 150.0 GiB) in 3s, read: 174.9 MiB/s, write: 120.7 MiB/s
INFO: 1% (1.6 GiB of 150.0 GiB) in 12s, read: 124.5 MiB/s, write: 115.1 MiB/s
INFO: 2% (3.1 GiB of 150.0 GiB) in 27s, read: 99.6 MiB/s, write: 98.9 MiB/s
INFO: 3% (4.6 GiB of 150.0 GiB) in 41s, read: 111.3 MiB/s, write: 107.0 MiB/s
INFO: 4% (6.1 GiB of 150.0 GiB) in 54s, read: 118.4 MiB/s, write: 117.6 MiB/s
INFO: 5% (7.6 GiB of 150.0 GiB) in 1m 7s, read: 120.2 MiB/s, write: 117.9 MiB/s
INFO: 6% (9.1 GiB of 150.0 GiB) in 1m 21s, read: 108.2 MiB/s, write: 107.5 MiB/s
INFO: 7% (10.6 GiB of 150.0 GiB) in 1m 34s, read: 115.3 MiB/s, write: 113.9 MiB/s
INFO: 8% (12.1 GiB of 150.0 GiB) in 1m 47s, read: 117.7 MiB/s, write: 115.9 MiB/s
INFO: 8% (12.8 GiB of 150.0 GiB) in 1m 55s, read: 98.1 MiB/s, write: 97.8 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: stopping kvm after backup task
ERROR: Backup of VM 107 failed - job failed with err -5 - Input/output error
INFO: Failed at 2021-02-17 15:55:33
INFO: Backup job finished with errors
TASK ERROR: job errors


Same issue, the VM is working fine. and no issue on VM!

The space use is 20GB and it's an windows server


Thanks

 

[oguz]

* does it work if you do the backup on local storage?

* can you access your synology from PVE normally?

* anything in journalctl while this is going on?

 

[CristianG]

Hi,

Thank you for your reply.

I did right now a backup on another VM

OUTPUT:

INFO: starting new backup job: vzdump 100 --remove 0 --compress zstd --node node1 --mode snapshot --storage Synology
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2021-02-17 16:57:57
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: firewall
INFO: include disk 'sata0' 'local-lvm:vm-100-disk-0' 50G
INFO: creating vzdump archive '/mnt/Synology/Proxmox/dump/vzdump-qemu-100-2021_02_17-16_57_57.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '8931f452-4ac3-4b97-81ed-442562d85ab4'
INFO: 9% (4.7 GiB of 50.0 GiB) in 3s, read: 1.6 GiB/s, write: 10.9 MiB/s
INFO: 18% (9.4 GiB of 50.0 GiB) in 6s, read: 1.6 GiB/s, write: 4.7 MiB/s
INFO: 21% (10.8 GiB of 50.0 GiB) in 9s, read: 456.1 MiB/s, write: 110.3 MiB/s
INFO: 28% (14.5 GiB of 50.0 GiB) in 12s, read: 1.2 GiB/s, write: 39.8 MiB/s
INFO: 38% (19.5 GiB of 50.0 GiB) in 15s, read: 1.7 GiB/s, write: 2.2 MiB/s
INFO: 48% (24.4 GiB of 50.0 GiB) in 18s, read: 1.6 GiB/s, write: 1.3 MiB/s
INFO: 58% (29.2 GiB of 50.0 GiB) in 21s, read: 1.6 GiB/s, write: 4.1 MiB/s
INFO: 62% (31.4 GiB of 50.0 GiB) in 24s, read: 751.2 MiB/s, write: 64.9 MiB/s
INFO: 67% (33.8 GiB of 50.0 GiB) in 27s, read: 818.0 MiB/s, write: 60.5 MiB/s
INFO: 75% (38.0 GiB of 50.0 GiB) in 30s, read: 1.4 GiB/s, write: 21.0 MiB/s
INFO: 76% (38.3 GiB of 50.0 GiB) in 33s, read: 108.7 MiB/s, write: 106.8 MiB/s
INFO: 77% (38.6 GiB of 50.0 GiB) in 36s, read: 108.8 MiB/s, write: 96.9 MiB/s
INFO: 78% (39.3 GiB of 50.0 GiB) in 39s, read: 239.2 MiB/s, write: 107.5 MiB/s
INFO: 79% (39.9 GiB of 50.0 GiB) in 42s, read: 211.5 MiB/s, write: 105.9 MiB/s
INFO: 81% (40.9 GiB of 50.0 GiB) in 45s, read: 323.9 MiB/s, write: 119.9 MiB/s
INFO: 91% (45.5 GiB of 50.0 GiB) in 48s, read: 1.5 GiB/s, write: 12.3 MiB/s
INFO: 100% (50.0 GiB of 50.0 GiB) in 51s, read: 1.5 GiB/s, write: 66.7 KiB/s
INFO: backup is sparse: 47.45 GiB (94%) total zero data
INFO: transferred 50.00 GiB in 51 seconds (1003.9 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 1.05GB
INFO: Finished Backup of VM 100 (00:00:54)
INFO: Backup finished at 2021-02-17 16:58:51
INFO: Backup job finished successfully
TASK OK

Accessing the Synology from PVE normally: YES




journalctl :


Feb 17 17:00:12 node1 pvedaemon[20810]: <root@pam> starting task UPID:node1:00006E16:00096534:602D4B9C:vzdump:107:root@pam:
Feb 17 17:00:12 node1 pvedaemon[28182]: INFO: starting new backup job: vzdump 107 --storage Synology --mode snapshot --node node1 --remove 0 --compress zstd
Feb 17 17:00:12 node1 pvedaemon[28182]: INFO: Starting Backup of VM 107 (qemu)
Feb 17 17:02:05 node1 kernel: ata9.00: exception Emask 0x0 SAct 0x100000 SErr 0x0 action 0x0
Feb 17 17:02:05 node1 kernel: ata9.00: irq_stat 0x40000008
Feb 17 17:02:05 node1 kernel: ata9.00: failed command: READ FPDMA QUEUED
Feb 17 17:02:05 node1 kernel: ata9.00: cmd 60/80:a0:80:bd:96/00:00:17:00:00/40 tag 20 ncq dma 65536 in
res 41/40:80:80:bd:96/00:00:17:00:00/40 Emask 0x409 (media error) <F>
Feb 17 17:02:05 node1 kernel: ata9.00: status: { DRDY ERR }
Feb 17 17:02:05 node1 kernel: ata9.00: error: { UNC }
Feb 17 17:02:05 node1 kernel: ata9.00: configured for UDMA/133
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 Sense Key : Medium Error [current]
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 Add. Sense: Unrecovered read error - auto reallocate failed
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 CDB: Read(10) 28 00 17 96 bd 80 00 00 80 00
Feb 17 17:02:05 node1 kernel: blk_update_request: I/O error, dev sda, sector 395754880 op 0x0READ) flags 0x0 phys_seg 15 prio class 0
Feb 17 17:02:05 node1 kernel: ata9: EH complete
Feb 17 17:02:05 node1 kernel: ata9.00: Enabling discard_zeroes_data
Feb 17 17:02:07 node1 pvedaemon[28182]: ERROR: Backup of VM 107 failed - job failed with err -5 - Input/output error
Feb 17 17:02:08 node1 pvedaemon[28182]: INFO: Backup job finished with errors
Feb 17 17:02:08 node1 pvedaemon[28182]: job errors
Feb 17 17:02:08 node1 pvedaemon[20810]: <root@pam> end task UPID:node1:00006E16:00096534:602D4B9C:vzdump:107:root@pam: job errors

 

[CristianG]

Anyone?

 

[oguz]

Feb 17 17:02:05 node1 kernel: ata9.00: exception Emask 0x0 SAct 0x100000 SErr 0x0 action 0x0
Feb 17 17:02:05 node1 kernel: ata9.00: irq_stat 0x40000008
Feb 17 17:02:05 node1 kernel: ata9.00: failed command: READ FPDMA QUEUED
Feb 17 17:02:05 node1 kernel: ata9.00: cmd 60/80:a0:80:bd:96/00:00:17:00:00/40 tag 20 ncq dma 65536 in
res 41/40:80:80:bd:96/00:00:17:00:00/40 Emask 0x409 (media error) <F>
Feb 17 17:02:05 node1 kernel: ata9.00: status: { DRDY ERR }
Feb 17 17:02:05 node1 kernel: ata9.00: error: { UNC }
Feb 17 17:02:05 node1 kernel: ata9.00: configured for UDMA/133
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 Sense Key : Medium Error [current]
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 Add. Sense: Unrecovered read error - auto reallocate failed
Feb 17 17:02:05 node1 kernel: sd 8:0:0:0: [sda] tag#20 CDB: Read(10) 28 00 17 96 bd 80 00 00 80 00
Feb 17 17:02:05 node1 kernel: blk_update_request: I/O error, dev sda, sector 395754880 op 0x0READ) flags 0x0 phys_seg 15 prio class 0
Feb 17 17:02:05 node1 kernel: ata9: EH complete
Feb 17 17:02:05 node1 kernel: ata9.00: Enabling discard_zeroes_data
Feb 17 17:02:07 node1 pvedaemon[28182]: ERROR: Backup of VM 107 failed - job failed with err -5 - Input/output error

looks like disk errors, your disk might be failing.

you could run smartctl -a /dev/sdb (and for the other disks) to see their health. i'd suggest you to also make backups of any important data in case the disk decides to fail

 

[CristianG]

Base by info :

=== START OF INFORMATION SECTION ===
Model Family: Intel 545s Series SSDs
Device Model: INTEL SSDSC2KW010T8
Serial Number: BTLA813400L71P0EGN
LU WWN Device Id: 5 5cd2e4 14f5249d1
Firmware Version: LHF0B2C
User Capacity: 1,024,209,543,168 bytes [1.02 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 (minor revision not indicated)
SATA Version is: SATA 3.2, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Thu Feb 18 21:55:54 2021 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED


SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0032 100 100 000 Old_age Always - 4
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1078
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 48
170 Unknown_Attribute 0x0033 099 099 010 Pre-fail Always - 0
171 Program_Fail_Count 0x0032 100 100 000 Old_age Always - 4
172 Erase_Fail_Count 0x0032 100 100 000 Old_age Always - 0
173 Unknown_Attribute 0x0033 100 100 005 Pre-fail Always - 17180721153
174 Unexpect_Power_Loss_Ct 0x0032 100 100 000 Old_age Always - 26
183 SATA_Downshift_Count 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 090 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 376
190 Temperature_Case 0x0032 029 051 000 Old_age Always - 29 (Min/Max 15/51)
192 Unsafe_Shutdown_Count 0x0032 100 100 000 Old_age Always - 26
199 CRC_Error_Count 0x0032 100 100 000 Old_age Always - 0
225 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 272043
226 Workld_Media_Wear_Indic 0x0032 100 100 000 Old_age Always - 0
227 Workld_Host_Reads_Perc 0x0032 100 100 000 Old_age Always - 0
228 Workload_Minutes 0x0032 100 100 000 Old_age Always - 0
232 Available_Reservd_Space 0x0033 099 099 010 Pre-fail Always - 0
233 Media_Wearout_Indicator 0x0032 099 099 000 Old_age Always - 0
236 Unknown_Attribute 0x0032 099 099 000 Old_age Always - 0
241 Host_Writes_32MiB 0x0032 100 100 000 Old_age Always - 272043
242 Host_Reads_32MiB 0x0032 100 100 000 Old_age Always - 224056
249 NAND_Writes_1GiB 0x0032 100 100 000 Old_age Always - 4752
252 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 4

SMART Error Log Version: 1
No Errors Logged



I believe something happen on VM side ! Any advice to repari, the VM start normally and work fine. only backup is not working !

Thank you