Snapshot problem

[Norman Uittenbogaart]

For some reason I was able to backup with snapshots before, but now it fails with the following error,

100: Mar 08 02:00:02 INFO: Starting Backup of VM 100 (openvz)
100: Mar 08 02:00:02 INFO: CTID 100 exist mounted running100: Mar 08 02:00:02 INFO: status = running100: Mar 08 02:00:03 INFO: backup mode: snapshot100: Mar 08 02:00:03 INFO: ionice priority: 7100: Mar 08 02:00:03 INFO: creating lvm snapshot of /dev/mapper/pve-data ('/dev/pve/vzsnap-node-0')100: Mar 08 02:00:03 INFO: /dev/mapper/pve-vzsnap--node--0: open failed: No such file or directory100: Mar 08 02:00:03 INFO: /dev/pve/vzsnap-node-0: not found: device not cleared100: Mar 08 02:00:03 INFO: Aborting. Failed to wipe snapshot exception store. 100: Mar 08 02:00:03 ERROR: Backup of VM 100 failed - command 'lvcreate --size 1024M --snapshot --name vzsnap-node-0 /dev/pve/data' failed: exit code 5

Could be maybe because one container ran out of memory before.
But how can I solve this ?

 

[dietmar]

Can you manually remove the snapshot:

# lvremove /dev/pve/vzsnap-node-0

Then try again.

 

[Norman Uittenbogaart]

strange thing is, it doesn't exist...

root@node:~# ls /dev/pve/
data  root  swap

and

root@node:~# lvremove /dev/pve/vzsnap-node-0  
One or more specified logical volume(s) not found.

It also doesnt seem to exist

root@node:~# lvdisplay -c
  /dev/pve/swap:pve:3:1:-1:1:10485760:1280:-1:0:-1:253:1
  /dev/pve/root:pve:3:1:-1:1:201326592:24576:-1:0:-1:253:0
  /dev/pve/data:pve:3:1:-1:1:1703936000:208000:-1:0:-1:253:2

So I don't see why it can't make it or clear it ......

 

[dietmar]

strange thing is, it doesn't exist...

That is OK so far. And you still get above error?

 

[Norman Uittenbogaart]

yes, same error ...

 

[dietmar]

yes, same error ...

Is this a standard installation - what kind of underlying storage do you use?

 

[Norman Uittenbogaart]

Its the beta 2 iso from the site here, no special configurations.
Storage is RAID 10 on a Adaptec 5405 card.

Only I added a backup directory on the Root volume for the snapshots....

 

[dietmar]

Its the beta 2 iso

Please can you update your system

# aptitude update
# aptitude full-upgrade

The reboot (new kernel) and test again.

 

[Norman Uittenbogaart]

Im running already running the new kernel,

Linux node 2.6.32-7-pve #1 SMP Thu Feb 16 09:00:32 CET 2012 x86_64 GNU/Linux

 

[dietmar]

Im running already running the new kernel,

Linux node 2.6.32-7-pve #1 SMP Thu Feb 16 09:00:32 CET 2012 x86_64 GNU/Linux

What is the output of

# pveversion -v

and

# pvesm status

 

[Norman Uittenbogaart]

root@node:~# pveversion -v
pve-manager: 2.0-38 (pve-manager/2.0/af81df02)
running kernel: 2.6.32-7-pve
proxmox-ve-2.6.32: 2.0-60
pve-kernel-2.6.32-7-pve: 2.6.32-60
lvm2: 2.02.88-2pve1
clvm: 2.02.88-2pve1
corosync-pve: 1.4.1-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-1
pve-cluster: 1.0-23
qemu-server: 2.0-25
pve-firmware: 1.0-15
libpve-common-perl: 1.0-17
libpve-access-control: 1.0-17
libpve-storage-perl: 2.0-12
vncterm: 1.0-2
vzctl: 3.0.30-2pve1
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-5
ksm-control-daemon: 1.1-1

root@node:~# pvesm status
Backup    dir 1        99083868        21228156        72822548 23.07%
local     dir 1       838600256        38808676       799791580 5.13%

 

[dietmar]

And what is the output of

# vgs

 

[Norman Uittenbogaart]

root@node:~# vgs  
VG   #PV #LV #SN Attr   VSize   VFree
  pve    1   3   0 wz--n- 929.50g 16.00g

 

[dietmar]

Please can you try to create a snapshot manually - what is the output of:

# lvcreate --size 1000M --snapshot --name vzsnap-node-0 /dev/pve/data

please remove the snapshot later with (if above command worked):

# lvremove /dev/pve/vzsnap-node-0

 

[Norman Uittenbogaart]

Hmm I get

root@node:~# lvcreate --size 1000M --snapshot --name vzsnap-node-0 /dev/pve/data  
/dev/mapper/pve-vzsnap--node--0: open failed: No such file or directory
  /dev/pve/vzsnap-node-0: not found: device not cleared
  Aborting. Failed to wipe snapshot exception store.

strange :S

 

[dietmar]

Is there any hint in /var/log/syslog?

 

[Norman Uittenbogaart]

none also not in messages ...

 

[dietmar]

can you create a normal LV?

# lvcreate --size 1000M --name atest pve
# lvremove /dev/pve/atest

 

[Norman Uittenbogaart]

Nope something got messed up ...

root@node:~# lvcreate --size 1000M --name atest pve
  /dev/mapper/pve-atest: open failed: No such file or directory
  /dev/pve/atest: not found: device not cleared
  Aborting. Failed to wipe start of new LV.

No clue how I can clear this tho :S

 

[Norman Uittenbogaart]

I found it !

Somehow because a OpenVZ ve ran out of its memory, it started shutting down processes ....
LVM seems to depend on udev which was not running ...

Thanks for the help dietmar !