lvremove failed - trying again in XX seconds

[lcaron]

Hi,

I'm sorry to heavily insist but I believe there is a bug (or maybe a
misunderstanding on my side) with vzdump/snapshots/lvm backups on 1.9.

I'm constantly getting "lvremove failed" while using snapshot backups.

My setup:
4 proxmox boxes (2.0-28):

# pveversion --verbose
pve-manager: 2.0-28 (pve-manager/2.0/30bb4d33)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 2.0-60
pve-kernel-2.6.32-6-pve: 2.6.32-55
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.88-2pve1
clvm: 2.02.88-2pve1
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-1
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-1
pve-cluster: 1.0-22
qemu-server: 2.0-17
pve-firmware: 1.0-15
libpve-common-perl: 1.0-14
libpve-access-control: 1.0-11
libpve-storage-perl: 2.0-11
vncterm: 1.0-2
vzctl: 3.0.30-2pve1
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-3
ksm-control-daemon: 1.1-1

root FS is on softraid. This is only the root FS.
# cat /proc/mdstat
md0 : active raid1 sda1[0] sdb1[1]
31254336 blocks [2/2] [UU]
bitmap: 1/1 pages [4KB], 65536KB chunk

# mount | grep pve
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)
filer-002:/mnt/vg_md2/backups_proxmox/export on /mnt/pve/backups_tmm type nfs (rw,vers=3,addr=192.168.0.64)

# cat 701.conf
boot: dcn
bootdisk: ide0
cores: 1
ide0: LUN_lcaron:vm-701-disk-1
ide2: local:iso/grml64_2011.05.iso,media=cdrom
memory: 2048
name: vm-lc
net0: e1000=12:C1:3B:28:55:3B,bridge=vmbr0
ostype: l26
sockets: 1

# lvdisplay
--- Logical volume ---
LV Name /dev/VG_FAS-2040-SIEGE-LCARON/vm-701-disk-1
VG Name VG_FAS-2040-SIEGE-LCARON
LV UUID XjqsmN-MGVL-JD5F-79ST-6LvI-YdLn-7otcGw
LV Write Access read/write
LV Status NOT available
LV Size 32.00 GiB
Current LE 8192
Segments 1
Allocation inherit
Read ahead sectors auto


# iscsiadm --mode node
10.100.0.10:3260,2002 iqn.1992-08.com.netapp:sn.142254160

# vzdump --mode snapshot --mailto admin-proxmox@lncsa.fr --all 1 --node proxmox-vty-001 --maxfiles 2 --storage backups_tmm
INFO: starting new backup job: vzdump --mailto admin-proxmox@lncsa.fr --mode snapshot --maxfiles 2 --storage backups_tmm --node proxmox-vty-001 --all 1
INFO: Starting Backup of VM 701 (qemu)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: Logical volume "vzsnap-proxmox-vty-001-0" created
INFO: creating archive '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_02_24-11_41_25.tar'
INFO: adding '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_02_24-11_41_25.tmp/qemu-server.conf' to archive ('qemu-server.conf')
INFO: adding '/dev/VG_FAS-2040-SIEGE-LCARON/vzsnap-proxmox-vty-001-0' to archive ('vm-disk-ide0.raw')
INFO: Total bytes written: 34359740928 (34.17 MiB/s)
INFO: archive file size: 32.00GB
INFO: delete old backup '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_02_21-12_45_02.tar.lzo'
INFO: lvremove failed - trying again in 8 seconds
INFO: lvremove failed - trying again in 16 seconds
INFO: lvremove failed - trying again in 32 seconds
INFO: Finished Backup of VM 701 (00:16:35)
INFO: Backup job finished successfully

The only way out is then a reboot of the host.

Suspend works well (just the VM is unavailable during backups).

The exact same setup works fine under ProxMox 1.8 (with kernel 2.6.35).

Regards,

Laurent

 

[tom]

pls do not never double posts - in the forum AND the mailing list. you just waste time of all people which could probably help here as almost all are subscribed on both channels.

also search the forum, this issue is already discussed several times. and finally, you run 2.0 and not 1.9 like you write in the beginning of your post.

 

[lcaron]

Thanks for your reply tom.
I unfortunately did search the mailing list archives, this forum without any real solution to this issue.

Regards,

Laurent

 

[dietmar]

does something access the mounted data at '/mnt/pve/backups_tmm/'. Try to use lsof or fuser to find out.

 

[lcaron]

Hi Dietmar,
# lsof | grep backups_tmm
# (no output)
# fuser /dev/VG_SSD_proxmox-vty-001/vzsnap-proxmox-vty-001-0
# (no output)

I would then assume nothing is keeping those open.

The backup jobs however ends in:

INFO: archive file size: 1.00GB
INFO: delete old backup '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_02_24-16_58_50.tar'
INFO: lvremove failed - trying again in 8 seconds
INFO: lvremove failed - trying again in 16 seconds
INFO: lvremove failed - trying again in 32 seconds
ERROR: command 'lvremove -f /dev/VG_SSD_proxmox-vty-001/vzsnap-proxmox-vty-001-0' failed: Not a CODE reference at /usr/share/perl5/PVE/Tools.pm line 356.
INFO: Finished Backup of VM 701 (00:02:19)
INFO: Backup job finished successfully

When trying to remove the snapshot:

# lvremove -f /dev/VG_SSD_proxmox-vty-001/vzsnap-proxmox-vty-001-0
Unable to deactivate open VG_SSD_proxmox--vty--001-vzsnap--proxmox--vty--001--0-cow (253:4)
Failed to resume vzsnap-proxmox-vty-001-0.
libdevmapper exiting with 1 device(s) still suspended.

Any access to LVM (lvs, lvdisplay, ...) hangs

 

[dietmar]

Hi Dietmar,
# lsof | grep backups_tmm

Please can you send the the output of 'lsof' (to <dietmar@proxmox.com>)

 

[lcaron]

Hi Dietmar,

Sent.

Thanks

Laurent

 

[dietmar]

Can you manually remobe the snapshot (lvremove)?

And please can you update to the latest version?

 

[lcaron]

I can manually remove the snapshot (lvremove -f ...)
I'm running the latest version with the same issues.

 

[dietmar]

ERROR: command 'lvremove -f /dev/VG_SSD_proxmox-vty-001/vzsnap-proxmox-vty-001-0' failed: Not a CODE reference at /usr/share/perl5/PVE/Tools.pm line 356.

I just uploaded a fix for that warning qemu-server 2.0-23

Do you always get that error, or only sometimes?

 

[dietmar]

I can manually remove the snapshot (lvremove -f ...)

But you first posted that you can't remove manually? So you can remove after some time? Or if you call lvremove more than once?

 

[lcaron]

Hi,

I did update to latest version and it seems it is fixed now:


701: Mar 01 10:35:36 INFO: Starting Backup of VM 701 (qemu) 701: Mar 01 10:35:36 INFO: status = running 701: Mar 01 10:35:37 INFO: backup mode: snapshot 701: Mar 01 10:35:37 INFO: ionice priority: 7 701: Mar 01 10:35:38 INFO: Logical volume "vzsnap-proxmox-vty-001-0" created 701: Mar 01 10:35:38 INFO: creating archive '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_03_01-10_35_35.tar' 701: Mar 01 10:35:38 INFO: adding '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_03_01-10_35_35.tmp/qemu-server.conf' to archive ('qemu-server.conf') 701: Mar 01 10:35:39 INFO: adding '/dev/VG_SSD_proxmox-vty-001/vzsnap-proxmox-vty-001-0' to archive ('vm-disk-ide0.raw') 701: Mar 01 10:36:13 INFO: Total bytes written: 1073744384 (29.26 MiB/s) 701: Mar 01 10:36:13 INFO: archive file size: 1.00GB 701: Mar 01 10:36:13 INFO: delete old backup '/mnt/pve/backups_tmm/dump/vzdump-qemu-701-2012_02_29-12_45_02.tar.lzo' 701: Mar 01 10:36:15 INFO: Finished Backup of VM 701 (00:00:40)