Backup of VE failed

[gkovacs]

This seems to happen to one VE running on a clustered proxmox server, since we upgraded to 1.4.

This is the log file created by the backup job:

Oct 23 07:50:37 INFO: Starting Backup of VM 104 (openvz)
Oct 23 07:50:37 INFO: CTID 104 exist mounted running
Oct 23 07:50:37 INFO: status = CTID 104 exist mounted running
Oct 23 07:50:38 INFO: backup mode: snapshot
Oct 23 07:50:38 INFO: bandwidth limit: 10240 KB/s
Oct 23 07:50:38 INFO: trying to remove stale snapshot '/dev/pve/vzsnap-proxmox2-0'
Oct 23 07:50:38 INFO: umount: /mnt/vzsnap0: device is busy
Oct 23 07:50:38 INFO: umount: /mnt/vzsnap0: device is busy
Oct 23 07:50:38 ERROR: command 'umount /mnt/vzsnap0' failed with exit code 1
Oct 23 07:50:38 INFO:   /dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Oct 23 07:50:38 INFO:   Can't remove open logical volume "vzsnap-proxmox2-0"
Oct 23 07:50:38 ERROR: command 'lvremove -f /dev/pve/vzsnap-proxmox2-0' failed with exit code 5
Oct 23 07:50:38 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox2-0')
Oct 23 07:50:38 INFO:   /dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Oct 23 07:50:38 INFO:   Logical volume "vzsnap-proxmox2-0" already exists in volume group "pve"
Oct 23 07:50:38 INFO:   /dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
Oct 23 07:50:38 INFO:   Can't remove open logical volume "vzsnap-proxmox2-0"
Oct 23 07:50:38 ERROR: command 'lvremove -f /dev/pve/vzsnap-proxmox2-0' failed with exit code 5
Oct 23 07:50:38 ERROR: Backup of VM 104 failed - command 'lvcreate --size 1024M --snapshot --name vzsnap-proxmox2-0 /dev/pve/data' failed with exit code 5

Any ideas?

 

[hisaltesse]

Run apt-get update and apt-get upgrade again. vzdump was just updated.

 

[gkovacs]

Actually, the problem is older than 1.4. I've found the logfile from last weekend's backup, which shows the same problem:

Oct 17 07:29:41 INFO: Starting Backup of VM 104 (openvz)
Oct 17 07:29:41 INFO: status = CTID 104 exist mounted running
Oct 17 07:29:41 INFO: [COLOR=Red]trying to remove stale snapshot '/dev/pve/vzsnap'[/COLOR]
[COLOR=Black]Oct 17 07:29:41 INFO: umount: /mnt/vzsnap: device is busy[/COLOR]
Oct 17 07:29:41 INFO: umount: /mnt/vzsnap: device is busy
Oct 17 07:29:41 INFO: [COLOR=Red]command 'umount /mnt/vzsnap' failed with exit code 1[/COLOR]
Oct 17 07:29:41 INFO:  [COLOR=Red] Can't remove open logical volume "vzsnap"[/COLOR]
Oct 17 07:29:41 INFO: [COLOR=Red]command '/sbin/lvremove -f /dev/pve/vzsnap' failed with exit code 5[/COLOR]
Oct 17 07:29:41 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap')
Oct 17 07:29:41 INFO:   Logical volume "vzsnap" already exists in volume group "pve"
Oct 17 07:29:41 ERROR: Backup of VM 104 failed - command '/sbin/lvcreate --size 1024M --snapshot --name vzsnap /dev/pve/data' failed with exit code 5

How can I remove the stale snapshot?
Should I force an fsck or is it an LVM internal problem?

Here is the output of a couple of LVM tools:

proxmox2:/backup/weekendly# vgs
File descriptor 7 left open
  /dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
  VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   4   1 wz--n- 148.55G 3.00G

proxmox2:/backup/weekendly# lvs
File descriptor 7 left open
[COLOR=Black]  /dev/dm-5: read failed after 0 of 4096 at 0: Input/output error
  LV     VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  data   pve  owi-ao 103.30G
  root   pve  -wi-ao  37.25G
  swap   pve  -wi-ao   4.00G
[/COLOR] [COLOR=Black]  vzsnap pve  Swi-I-   1.00G data   100.00[/COLOR]

proxmox2:/backup/weekendly# lvscan
File descriptor 7 left open
[COLOR=Red]  /dev/dm-5: read failed after 0 of 4096 at 0: Input/output error[/COLOR]
  ACTIVE            '/dev/pve/swap' [4.00 GB] inherit
  ACTIVE            '/dev/pve/root' [37.25 GB] inherit
  inactive Original '/dev/pve/data' [103.30 GB] inherit
[COLOR=Red]  inactive Snapshot '/dev/pve/vzsnap' [1.00 GB] inherit[/COLOR]

What is /dev/dm-5? Is it related to the snapshot?

 

[dietmar]

That I/O Error looks quite strange. Anyways, try to remove the snapshot manually with 'lvremove'

 

[dietmar]

Ah, got it now. Your snapshot is simple full (100% usage). Please try to increase snapshot size in (/etc/vzdump.conf - for example set "size: 2048")