VZdump issue after replacing drive in raid

[commgdog]

Hello,

I have ran into an issue recently that I haven't been able so find a solution on (Google seems to turn up no results).

My node has 4 VM's (2 Windows, 2 Linux) that are on an LVM storage made up of a raid 10 array. One of my drives had failed and I replaced it, rebuilding went fine and everything checked out.

Each night I have vzdump run to do backups of the 4 VM's. The backups have been working great for the last two years, but now one of them is not able to back up, and I get the following error after about 30% on the backup:

Aug 29 08:52:47 INFO: status: 32% (103678345216/322122547200), sparse 4% (14100836352), duration 2615, 241/0 MB/s
Aug 29 08:52:47 ERROR: job failed with err -5 - Input/output error
Aug 29 08:52:47 INFO: aborting backup job
Aug 29 08:52:50 ERROR: Backup of VM 103 failed - job failed with err -5 - Input/output error

I am using the following command to run the backup (scheduled via the GUI)

vzdump 103 --storage sv-01-10-bak --quiet 1 --compress lzo --node sv-01-08 --mode snapshot

Where sv-01-10-bak is my backup storage.

I have searched through the vzdump man pages and google and can't seem to find out what the error means.

Any Idea on what could be causing the issue?

[Edit] The VM in question is a Windows 7 VM

 

[LnxBil]

What about your system log, e.g. dmesg and /var/log/syslog around the time in question?

 

[commgdog]

What about your system log, e.g. dmesg and /var/log/syslog around the time in question?

Here is what I found:

Aug 30 04:28:33 sv-01-08 kernel: [196638.225250] sd 0:1:0:0: [sdb] tag#0 CDB: Read(10) 28 00 3e 11 9d 00 00 00 80 00
Aug 30 04:28:33 sv-01-08 kernel: [196638.225253] mpt2sas_cm0:   volume wwid(0x0197e32b77211dc3)
Aug 30 04:28:33 sv-01-08 kernel: [196638.225254] mpt2sas_cm0:   handle(0x0143), ioc_status(success)(0x0000), smid(71)
Aug 30 04:28:33 sv-01-08 kernel: [196638.225255] mpt2sas_cm0:   request_len(65536), underflow(65536), resid(65536)
Aug 30 04:28:33 sv-01-08 kernel: [196638.225256] mpt2sas_cm0:   tag(0), transfer_count(0), sc->result(0x00000000)
Aug 30 04:28:33 sv-01-08 kernel: [196638.225257] mpt2sas_cm0:   scsi_status(check condition)(0x02), scsi_state(autosense valid )(0x01)
Aug 30 04:28:33 sv-01-08 kernel: [196638.225258] mpt2sas_cm0:   [sense_key,asc,ascq]: [0x03,0x11,0x00], count(16)
Aug 30 04:28:33 sv-01-08 kernel: [196638.225266] sd 0:1:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Aug 30 04:28:33 sv-01-08 kernel: [196638.225269] sd 0:1:0:0: [sdb] tag#0 Sense Key : Medium Error [current]
Aug 30 04:28:33 sv-01-08 kernel: [196638.225271] sd 0:1:0:0: [sdb] tag#0 Add. Sense: Unrecovered read error
Aug 30 04:28:33 sv-01-08 kernel: [196638.225272] sd 0:1:0:0: [sdb] tag#0 CDB: Read(10) 28 00 3e 11 9d 00 00 00 80 00
Aug 30 04:28:33 sv-01-08 kernel: [196638.225274] blk_update_request: critical medium error, dev sdb, sector 1041341696

Running my sas raid utility shows that my raid is having issues with a different drive, that may be what is causing it. I plan on moving the 4 VMs on the machine to a different node, but this one VM is still giving my trouble. I am not able to dump it and then restore on the other node, yet the VM boots fine and runs fine (as far as I am aware).

Any thoughts on getting the VM to dump properly?

 

[LnxBil]

It seams that your disk has problems. If you see errors in the OS (technically after the raid controller), then you're probably in big trouble. It means that the errors propagate through the raid controller to your OS (mostly errors on multiple disks).

Without a working backup your data is gone, sorry.