failing backups

[jkister]

I noticed my backups recently started failing:

INFO: starting new backup job: vzdump --quiet 1 --mailto  proxmox@example.com --mode snapshot --compress gzip --storage  backup0 --all 1
INFO: Starting Backup of VM 100 (openvz)
INFO: CTID 100 exist mounted running
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: trying to remove stale snapshot '/dev/pve/vzsnap-proxmox1-0'
INFO: umount: /mnt/vzsnap0: not mounted
ERROR: command 'umount /mnt/vzsnap0' failed: exit code 1
INFO:   /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771608576: Input/output error
INFO:   /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771665920: Input/output error
INFO:   /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 0: Input/output error
INFO:   /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 4096: Input/output error
INFO:   Logical volume "vzsnap-proxmox1-0" successfully removed
INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-proxmox1-0')
INFO:   Logical volume "vzsnap-proxmox1-0" created
INFO: creating archive '/var/lib/backup0/dump/vzdump-openvz-100-2013_08_10-03_00_02.tar.gz'
INFO: gzip: stdout: No space left on device
INFO: lvremove failed - trying again in 8 seconds
INFO: lvremove failed - trying again in 16 seconds
INFO: lvremove failed - trying again in 32 seconds
ERROR: command 'lvremove -f /dev/pve/vzsnap-proxmox1-0' failed: exit code 5
ERROR:  Backup of VM 100 failed - command '(cd /mnt/vzsnap0/private/100;find .  '(' -regex '^\.$' ')' -o '(' -type 's' -prune ')' -o -print0|sed  's/\\/\\\\/g'|tar cpf - --totals --sparse --numeric-owner --no-recursion  --one-file-system --null -T -|gzip)  >/var/lib/backup0/dump/vzdump-openvz-100-2013_08_10-03_00_02.tar.dat'  failed: exit code 1
INFO: Backup job finished with errors
TASK ERROR: job errors

This may have to do with me recently upgrading to proxmox3. or maybe it's just a coincidence.

I have confirmed that /mnt/vzsnap0 is a directory and nothing is mounted on it.

was something managing my backups by removing old dumps that is broken now? or was this just a coincidence that my backups are breaking because of space issues ?

 

[tom]

we have new LVM packages which should fix the issue.
(currently in pvetest repo only).

 

[deltaend]

we have new LVM packages which should fix the issue.
(currently in pvetest repo only).

Will this also fix the crazy slow backups that we run into? Our problem may be different, but we are backing up to an NFS volume and it works for *some* linux virtual machines, and even for CentOS machines, but for this particular one, it slows down to a crawl within the first 3% of the backup and locks up both the PM node and the virtual machine itself. It sends bursts of data and then sits for a long time and then sends another burst. There doesn't seem to be an errors being generated when this, it just locks everything up.

A couple of things to note, other virtual machines on this node, also running CentOS do not have this problem. It is the same problem no matter if we have the drive as QCOW2 or VMKD. This virtual machine has a large amount of memory, larger than any other virtual machine on that node (16GB).

Any thoughts?

 

[jkister]

we have new LVM packages which should fix the issue.
(currently in pvetest repo only).

I added pvetest to /etc/apt/sources.list
apt-get update
apt-get install lvm2

Setting up LVM Volume Groups...  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771608576: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771665920: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 4096: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771608576: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771665920: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 4096: Input/output error
done.
update-initramfs: deferring update (trigger activated)
Setting up clvm (2.02.98-pve4) ...
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-20-pve

What can I do to fix this up?

root@proxmox1:/dev# cd /dev/pve
root@proxmox1:/dev/pve# ls -al
total 0
drwxr-xr-x  2 root root  120 Aug 10 03:00 .
drwxr-xr-x 17 root root 6040 Jun 22 03:00 ..
lrwxrwxrwx  1 root root    7 Aug 10 03:00 data -> ../dm-2
lrwxrwxrwx  1 root root    7 Jun 20 22:38 root -> ../dm-0
lrwxrwxrwx  1 root root    7 Jun 20 22:39 swap -> ../dm-1
lrwxrwxrwx  1 root root    7 Aug 10 03:00 vzsnap-proxmox1-0 -> ../dm-3
root@proxmox1:/dev/pve# ls -la ../dm-3
brw-rw---T 1 root disk 253, 3 Aug 10 03:00 ../dm-3

 

[dietmar]

Try to remove that stale snapshot manually:

# lvremove /dev/pve/vzsnap-proxmox1-0

 

[jkister]

Try to remove that stale snapshot manually:

# lvremove /dev/pve/vzsnap-proxmox1-0

Perfect.

root@proxmox1:~# lvremove /dev/pve/vzsnap-proxmox1-0 
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771608576: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 184771665920: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 0: Input/output error
  /dev/pve/vzsnap-proxmox1-0: read failed after 0 of 4096 at 4096: Input/output error
Do you really want to remove and DISCARD active logical volume vzsnap-proxmox1-0? [y/n]: y
  Logical volume "vzsnap-proxmox1-0" successfully removed
root@proxmox1:~# apt-get install lvm2  --reinstall
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 25 not upgraded.
Need to get 0 B/612 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 40933 files and directories currently installed.)
Preparing to replace lvm2 2.02.98-pve4 (using .../lvm2_2.02.98-pve4_amd64.deb) ...
Unpacking replacement lvm2 ...
Processing triggers for man-db ...
Setting up lvm2 (2.02.98-pve4) ...
Setting up LVM Volume Groups...done.
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools ...
update-initramfs: Generating /boot/initrd.img-2.6.32-20-pve

Thanks for your help!

 

[DANIEL ALVARADO]

I have tired of installing proxmox on my PowerEdge R710 Xeon x5570 Quad Core 2.93GhZ server and I can not find a solution to the problem, someone knows if I can solve this!

 

[sb-jw]

@DANIEL ALVARADO why you take over a thread with an completely different topic? Please open an new thread!

Btw. it seems the /dev/sda is broken, check it. If you want to answer me, open a new thread first and don't post again in this topic.

 

[oguz]

closed thread. please do not necro posts from 2013 with irrelevant problems.