cant kill vzdump precess

[blackjack]

I ran backup in shapshot mode and it hang proxmox 1.9 gui now. I use NFS share storage. So i tried to kill vzdump process. But one process I couldnot kill. The web GUI is still hanging.

ps -ef|grep vzdump
root      324654       1 87 Mar08 ?        3-06:41:37 /usr/lib/qemu-server/vmtar /mnt/pve/storage/vzdump-qemu-103-2012_03_08-02_00_02.tmp/qemu-server.conf qemu-server.conf /mnt/vzsnap0/images/103/vm-disk-ide0.qcow2 vm-disk-ide0.qcow2

First i have unmount the /mnt/vzsnap0

umount -l /mnt/vzsnap0

Then ,I tried lvremove and dmsetup remove

lvremove -f /dev/pve/vzsnap-VMhost1-0
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  Can't remove open logical volume "vzsnap-VMhost1-0"

dmsetup remove /dev/mapper/pve-vzsnap--VMhost1--0
device-mapper: remove ioctl failed: Device or resource busy
Command failed

dmsetup remove /dev/mapper/pve-vzsnap--VMhost1--0-cow
device-mapper: remove ioctl failed: Device or resource busy
Command failed

dev/mapper shows

/dev/mapper# ls
control   pve-data-real  pve-storage  pve-vzsnap--VMhost1--0
pve-data  pve-root     pve-swap     pve-vzsnap--VMhost1--0-cow

lvscan

lvscan
  /dev/dm-2: read failed after 0 of 4096 at 0: Input/output error
  ACTIVE            '/dev/pve/swap' [4.00 GB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GB] inherit
  inactive Original '/dev/pve/data' [361.26 GB] inherit
  ACTIVE            '/dev/pve/storage' [200.00 GB] inherit
  inactive Snapshot '/dev/pve/vzsnap-VMhost1-0' [1.00 GB] inherit

log file

Mar 08 02:00:02 INFO: Starting Backup of VM 103 (qemu)
Mar 08 02:00:02 INFO: running
Mar 08 02:00:02 INFO: status = running
Mar 08 02:00:03 INFO: backup mode: snapshot
Mar 08 02:00:03 INFO: ionice priority: 7
Mar 08 02:00:03 INFO:   Logical volume "vzsnap-VMhost1-0" created
Mar 08 02:00:03 INFO: creating archive '/mnt/pve/storage/vzdump-qemu-103-2012_03_08-02_00_02.tar'

dmsetup table

pve-vzsnap--VMhost1--0-cow: 0 2097152 linear 8:2 419430784
pve-data-real: 0 757620736 linear 8:18 209715584
pve-swap: 0 8388608 linear 8:18 384
pve-root: 0 201326592 linear 8:18 8388992
pve-data: 0 757620736 snapshot-origin 253:4
pve-vzsnap--VMhost1--0: 0 757620736 snapshot 253:4 253:5 P 8
pve-storage: 0 419430400 linear 8:2 384

mount

/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
procbususb on /proc/bus/usb type usbfs (rw)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sdb1 on /boot type ext3 (rw)
/dev/mapper/pve-storage on /storage type ext3 (rw)
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)
xx.xx.xx.xx:/storage on /mnt/pve/storage type nfs (rw,addr=xx.xx.xx.xx)

how to get out of this problem ??? Probably the /mnt/pve/storage is busy . Plez give me a solution.

 

[dietmar]

Probably the /mnt/pve/storage is busy.

Behaves like your NFS server is not responding? Please check if the server is working correctly.

Note: You need to get the NFS share working, else you need to reboot the node to get a clean state.

 

[dietmar]

BTW, using low level commands and remove active LVs is usually a bad idea (until you have a good reason to do that).

 

[blackjack]

Behaves like your NFS server is not responding? Please check if the server is working correctly.

Note: You need to get the NFS share working, else you need to reboot the node to get a clean state.

about rebooting , is there any command to do that. I have tried reboot on my proxmox node and it doesn't work. I do not have access to the physical hardware.

 

[dietmar]

about rebooting , is there any command to do that. I have tried reboot on my proxmox node and it doesn't work.

So maybe some shutdown script wait for NFS to. If so you need to power cycle the node.

 

[blackjack]

only restart help this issue. After restart I was able to use lvremove and remove the logical volume. then restart the NFS share and restart the pvedaemon on the other pve node. also unlock the VM which was being backing up. Now everything works fine. Now I am confused about how to take backup of my VM. I am planning to copy the qcow2 images to a directory in the pve server, so that I can restore it later on.

 

[dietmar]

What happened to your NFS server? Any hints in the NFS server logs?

 

[blackjack]

Sorry I do not know where to look for NFS server logs. But I checked the syslog and there is no information. In my case the PVE server do not have enough memory to run snapshot backup. So It hang the web GUI. A good idea in this case is to stop the VM you want to backup and take backup in stop mode.

My nfs server and storage is located on server 1. Server 1 and 2 backup via nfs share to server 1. It would be nice if there will be a fix, when system hangs in these situations.