Backup of VM failed - backup_complete_cb -5

[ozgurerdogan]

INFO: starting new backup job: vzdump 301 --remove 0 --mode snapshot --compress lzo --storage YEDEKLER --node vztl3
INFO: Starting Backup of VM 301 (qemu)
INFO: status = running
INFO: update VM 301: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/BACKUPS/dump/vzdump-qemu-301-2013_10_18-17_13_19.vma.lzo'
INFO: started backup task 'a8432414-9453-45de-8d0c-a96251d0b19a'
INFO: status: 0% (25034752/161061273600), sparse 0% (1449984), duration 3, 8/7 MB/s
INFO: status: 1% (1632567296/161061273600), sparse 0% (17092608), duration 62, 27/26 MB/s
INFO: status: 2% (3225419776/161061273600), sparse 0% (28073984), duration 104, 37/37 MB/s
INFO: status: 3% (4867686400/161061273600), sparse 0% (43896832), duration 182, 21/20 MB/s
INFO: status: 4% (6446514176/161061273600), sparse 0% (75808768), duration 224, 37/36 MB/s
INFO: status: 4% (7417167872/161061273600), sparse 0% (83787776), duration 267, 22/22 MB/s
ERROR: backup_complete_cb -5
INFO: aborting backup job
ERROR: Backup of VM 301 failed - backup_complete_cb -5
INFO: Backup job finished with errors
TASK ERROR: job errors

I restarted VM and node but did not help.

 

[dietmar]

Try to test the VM disk images with 'qemu-img check'

 

[ozgurerdogan]

Try to test the VM disk images with 'qemu-img check'

This is raw and I get:
root@vztl3:/var/lib/vz/images/301# qemu-img check vm-301-disk-1.raw
qemu-img: This image format does not support checks

 

[ozgurerdogan]

When this error occurs, I see;


Oct 18 18:43:38 vztl3 kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 18:43:39 vztl3 kernel: ata3.00: BMDMA stat 0x25
Oct 18 18:43:39 vztl3 kernel: ata3.00: failed command: READ DMA EXT
Oct 18 18:43:39 vztl3 kernel: ata3.00: cmd 25/00:80:88:4c:6b/00:00:32:00:00/e0 tag 0 dma 65536 in
Oct 18 18:43:39 vztl3 kernel: res 51/40:1f:df:4c:6b/40:00:32:00:00/e0 Emask 0x9 (media error)
Oct 18 18:43:39 vztl3 kernel: ata3.00: status: { DRDY ERR }
Oct 18 18:43:39 vztl3 kernel: ata3.00: error: { UNC }
Oct 18 18:43:39 vztl3 kernel: ata3.00: configured for UDMA/133
Oct 18 18:43:39 vztl3 kernel: ata3: EH complete
Oct 18 18:43:39 vztl3 kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 18:43:39 vztl3 kernel: ata3.00: BMDMA stat 0x25
Oct 18 18:43:39 vztl3 kernel: ata3.00: failed command: READ DMA EXT
Oct 18 18:43:39 vztl3 kernel: ata3.00: cmd 25/00:80:88:4c:6b/00:00:32:00:00/e0 tag 0 dma 65536 in
Oct 18 18:43:39 vztl3 kernel: res 51/40:1f:df:4c:6b/40:00:32:00:00/e0 Emask 0x9 (media error)
Oct 18 18:43:39 vztl3 kernel: ata3.00: status: { DRDY ERR }
Oct 18 18:43:39 vztl3 kernel: ata3.00: error: { UNC }
Oct 18 18:43:39 vztl3 kernel: ata3.00: configured for UDMA/133
Oct 18 18:43:39 vztl3 kernel: ata3: EH complete
Oct 18 18:43:39 vztl3 kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 18:43:39 vztl3 kernel: ata3.00: BMDMA stat 0x25
Oct 18 18:43:39 vztl3 kernel: ata3.00: failed command: READ DMA EXT
Oct 18 18:43:39 vztl3 kernel: ata3.00: cmd 25/00:80:88:4c:6b/00:00:32:00:00/e0 tag 0 dma 65536 in
Oct 18 18:43:39 vztl3 kernel: res 51/40:1f:df:4c:6b/40:00:32:00:00/e0 Emask 0x9 (media error)
Oct 18 18:43:39 vztl3 kernel: ata3.00: status: { DRDY ERR }
Oct 18 18:43:39 vztl3 kernel: ata3.00: error: { UNC }
Oct 18 18:43:39 vztl3 kernel: ata3.00: configured for UDMA/133
Oct 18 18:43:39 vztl3 kernel: ata3: EH complete
Oct 18 18:43:39 vztl3 kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 18:43:39 vztl3 kernel: ata3.00: BMDMA stat 0x25
Oct 18 18:43:39 vztl3 kernel: ata3.00: failed command: READ DMA EXT
Oct 18 18:43:39 vztl3 kernel: ata3.00: cmd 25/00:80:88:4c:6b/00:00:32:00:00/e0 tag 0 dma 65536 in
Oct 18 18:43:39 vztl3 kernel: res 51/40:1f:df:4c:6b/40:00:32:00:00/e0 Emask 0x9 (media error)
Oct 18 18:43:39 vztl3 kernel: ata3.00: status: { DRDY ERR }
Oct 18 18:43:39 vztl3 kernel: ata3.00: error: { UNC }
Oct 18 18:43:39 vztl3 kernel: ata3.00: configured for UDMA/133
Oct 18 18:43:39 vztl3 kernel: ata3: EH complete
Oct 18 18:43:39 vztl3 kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 18:43:39 vztl3 kernel: ata3.00: BMDMA stat 0x25
Oct 18 18:43:39 vztl3 kernel: ata3.00: failed command: READ DMA EXT
Oct 18 18:43:39 vztl3 kernel: ata3.00: cmd 25/00:80:88:4c:6b/00:00:32:00:00/e0 tag 0 dma 65536 in
Oct 18 18:43:39 vztl3 kernel: res 51/40:1f:df:4c:6b/40:00:32:00:00/e0 Emask 0x9 (media error)
Oct 18 18:43:39 vztl3 kernel: ata3.00: status: { DRDY ERR }
Oct 18 18:43:39 vztl3 kernel: ata3.00: error: { UNC }
Oct 18 18:43:39 vztl3 kernel: ata3.00: configured for UDMA/133
Oct 18 18:43:39 vztl3 kernel: ata3: EH complete
Oct 18 18:43:39 vztl3 kernel: ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Oct 18 18:43:39 vztl3 kernel: ata3.00: BMDMA stat 0x25
Oct 18 18:43:39 vztl3 kernel: ata3.00: failed command: READ DMA EXT
Oct 18 18:43:39 vztl3 kernel: ata3.00: cmd 25/00:80:88:4c:6b/00:00:32:00:00/e0 tag 0 dma 65536 in
Oct 18 18:43:39 vztl3 kernel: res 51/40:1f:df:4c:6b/40:00:32:00:00/e0 Emask 0x9 (media error)
Oct 18 18:43:39 vztl3 kernel: ata3.00: status: { DRDY ERR }
Oct 18 18:43:39 vztl3 kernel: ata3.00: error: { UNC }
Oct 18 18:43:39 vztl3 kernel: ata3.00: configured for UDMA/133
Oct 18 18:43:39 vztl3 kernel: sd 2:0:0:0: [sda] Unhandled sense code
Oct 18 18:43:39 vztl3 kernel: sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Oct 18 18:43:39 vztl3 kernel: sd 2:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
Oct 18 18:43:39 vztl3 kernel: Descriptor sense data with sense descriptors (in hex):
Oct 18 18:43:39 vztl3 kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Oct 18 18:43:39 vztl3 kernel: 32 6b 4c df
Oct 18 18:43:39 vztl3 kernel: sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
Oct 18 18:43:39 vztl3 kernel: sd 2:0:0:0: [sda] CDB: Read(10): 28 00 32 6b 4c 88 00 00 80 00
Oct 18 18:43:39 vztl3 kernel: ata3: EH complete

in syslog.

 

[dietmar]

Maybe your harddisk is damaged (media error)?

 

[ozgurerdogan]

Do not think so. It might be a kernel error. But will go with badblock check and see.

 

[ozgurerdogan]

Ok I noted that If I choose compress type .gz instead of .lzo, it can backup fine. But not with lzo. Do you think I should still suspect of bad disk?

 

[dietmar]

Do you think I should still suspect of bad disk?

I would check/test that disk at least.

 

[ozgurerdogan]

I would check/test that disk at least.

Confirmed that was a disk issue. After fix it is fine now. Thanks

 

[m.ardito]

Hi, do you remember which kind of issue it was (host disk, vm disk) and how did you fix it?
I'm eperiencing similar issues on a vm with disks over iscsi (no disk files to check with qemu-img)

Marco