vm locked and unlockable after a failed backup due to disk full

M

mmenaz

Renowned Member
Jun 25, 2009
835
25
93
Northern east Italy
Hi, in my home's Proxmox server I've few space left, so the scheduled backup fails.
This is not a problem since I've a test VM I'm going to remove soon.
But I've discovered that when VM 106 backup fails, the vm is locked and can't be unlocked! If I shutdown it and restart, it does not start because of the lock.
So the only solution so far is reboot the entire Proxmox server.
Here the details, what scares me is the Input/output error if I try to remove the lock file manually, and the fact that qm unlock tries to remove a file that does not exist.
Code: 
root@proxmox:/srv/backup/dump# cat vzdump-qemu-106-2013_01_06-04_04_57.log
Jan 06 04:04:57 INFO: Starting Backup of VM 106 (qemu)
Jan 06 04:04:57 INFO: status = running
Jan 06 04:04:58 INFO: backup mode: snapshot
Jan 06 04:04:58 INFO: ionice priority: 7
Jan 06 04:04:58 INFO:   Logical volume "vzsnap-proxmox-0" created
Jan 06 04:04:58 INFO: creating archive '/srv/backup/dump/vzdump-qemu-106-2013_01_06-04_04_57.tar.lzo'
Jan 06 04:04:58 INFO: adding '/srv/backup/dump/vzdump-qemu-106-2013_01_06-04_04_57.tmp/qemu-server.conf' to archive ('qemu-server.conf')
Jan 06 04:04:58 INFO: adding '/mnt/vzsnap0/images/106/vm-106-disk-1.qcow2' to archive ('vm-disk-ide0.qcow2')
Jan 06 04:25:28 INFO: lzop: No space left on device: <stdout>
Jan 06 04:25:28 INFO: received signal - terminate process
Jan 06 04:25:30 INFO: unable to open file '/etc/pve/nodes/proxmox/qemu-server/106.conf.tmp.118536' - Input/output error
Jan 06 04:25:31 INFO: error writing /etc/mtab.tmp: No space left on device
Jan 06 04:25:31 ERROR: command 'umount /mnt/vzsnap0' failed: exit code 16
Jan 06 04:25:39 INFO: lvremove failed - trying again in 8 seconds
Jan 06 04:25:47 INFO: lvremove failed - trying again in 16 seconds
Jan 06 04:26:03 INFO: lvremove failed - trying again in 32 seconds
Jan 06 04:26:39 ERROR: Backup of VM 106 failed - command '/usr/lib/qemu-server/vmtar  '/srv/backup/dump/vzdump-qemu-106-2013_01_06-04_04_57.tmp/qemu-server.conf' 'qemu-server.conf' '/mnt/vzsnap0/images/106/vm-106-disk-1.qcow2' 'vm-disk-ide0.qcow2'|lzop >/srv/backup/dump/vzdump-qemu-106-2013_01_06-04_04_57.tar.dat' failed: exit code 1
root@proxmox:/srv/backup/dump# 

root@proxmox:/srv/backup/dump# mount
/dev/mapper/pve-root on / type ext3 (rw,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
udev on /dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620)
/dev/mapper/pve-data on /var/lib/vz type ext3 (rw)
/dev/sda1 on /boot type ext3 (rw)
fusectl on /sys/fs/fuse/connections type fusectl (rw)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,default_permissions,allow_other)
beancounter on /proc/vz/beancounter type cgroup (rw,name=beancounter)
container on /proc/vz/container type cgroup (rw,name=container)
fairsched on /proc/vz/fairsched type cgroup (rw,name=fairsched)
root@proxmox:/srv/backup/dump# 

root@proxmox:/srv/backup/dump# cat /etc/pve/storage.cfg 
dir: local
        path /var/lib/vz
        content images,iso,vztmpl,rootdir

dir: backup
        path /srv/backup
        content backup
        maxfiles 2

root@proxmox:/srv/backup/dump#

root@proxmox:/srv/backup/dump# vgdisplay 
  --- Volume group ---
  VG Name               pve
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  484
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               372.11 GiB
  PE Size               4.00 MiB
  Total PE              95260
  Alloc PE / Size       91165 / 356.11 GiB
  Free  PE / Size       4095 / 16.00 GiB
  VG UUID               MdGnNc-bXFN-FY8X-BbTU-g7LB-JxrD-Kzr6ge
   
root@proxmox:/srv/backup/dump# 

root@proxmox:/etc/pve/nodes/proxmox/qemu-server# ls -l
total 5
-rw-r----- 1 root www-data  279 Jan  6 04:04 100.conf
-rw-r----- 1 root www-data  435 Apr 23  2012 101.conf
-rw-r----- 1 root www-data 1419 Dec 13 14:35 102.conf
-rw-r----- 1 root www-data  739 Mar 31  2012 104.conf
-rw-r----- 1 root www-data  489 Jan  6 04:04 106.conf
-rw-r----- 1 root www-data    0 Jan  6 04:25 106.conf.tmp.118536
-rw-r----- 1 root www-data    0 Dec 30 04:27 106.conf.tmp.314414
-rw-r----- 1 root www-data  262 Apr  9  2012 111.conf
-rw-r----- 1 root www-data  248 Mar 31  2012 114.conf
-rw-r----- 1 root www-data  239 Dec 20 21:37 200.conf
-rw-r----- 1 root www-data  906 Dec 28 00:31 204.conf

root@proxmox:/etc/pve/nodes/proxmox/qemu-server# rm 106.conf.tmp.314414
rm: cannot remove `106.conf.tmp.314414': Input/output error
root@proxmox:/etc/pve/nodes/proxmox/qemu-server# rm 106.conf.tmp.118536
rm: cannot remove `106.conf.tmp.118536': Input/output error
root@proxmox:/etc/pve/nodes/proxmox/qemu-server# qm unlock 106
unable to open file '/etc/pve/nodes/proxmox/qemu-server/106.conf.tmp.164466' - Input/output error
root@proxmox:/etc/pve/nodes/proxmox/qemu-server# 

root@proxmox:/var/lib/vz# ls -l /var/lib/vz/lock
total 0

root@proxmox:/srv/backup/dump# lsof | grep 106.conf
root@proxmox:/srv/backup/dump# 

root@proxmox:/srv/backup/dump# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/pve-root   92G   74G   14G  85% /
tmpfs                 3.9G     0  3.9G   0% /lib/init/rw
udev                  3.9G  228K  3.9G   1% /dev
tmpfs                 3.9G   19M  3.9G   1% /dev/shm
/dev/mapper/pve-data  253G   91G  162G  36% /var/lib/vz
/dev/sda1             495M  133M  338M  29% /boot
/dev/fuse              30M   20K   30M   1% /etc/pve
root@proxmox:/srv/backup/dump#
Is it a bug to report in bugtracker?
 
if your root filesystem is full (e.g. INFO: error writing /etc/mtab.tmp: No space left on device) nothing will work.
 
Hi Tom, my root file system WAS full, now is not since probably backup cleaned up some stuff after the error, never less I can't recover from this situation except with reboot.
Is it normal? I know you should never find yourself in a / full situation, but seems it can happen and would love to know if vzdump could do a better job restoring a working situation.
In addition, is not a big problem for me to reboot since is my home Proxmox, but would like to know if I could recover this problem without reboot.
Thanks a lot
 
Sure, as you can see in the long list of stuff I've provided and what I write in my message ("...that qm unlock tries to remove a file that does not exist").
 
Also after you have regained more free space? Maybe restarting pvedaemon after regained space will restore your cluster in a healthy state?
 
I've got this problem too... A backup failed due to lack of free space on the device, and now the VM it was backing up at the time won't unlock.
I tried "qm unlock <vmid>" but it gave an IO error.

Seems like a bug to me - it shouldn't require a reboot to get out of a failed backup.
 
There is.... Now. On both the backup drive and the host root.
But the lock remains and can't be unlocked.

This means no stopping, starting or backing up of the VM in question until the entire host is rebooted... That to me is a bug (and a pain in the rear end too).
 
tin said:
This means no stopping, starting or backing up of the VM in question until the entire host is rebooted... That to me is a bug (and a pain in the rear end too).
Click to expand...

There is by sure no need to reboot. You just need to restart some services (pve-cluster, pvestatd, apache2).
 
  • Like
Reactions: MarcoP
Is there an equivalent unlock command for OpenVZ containers?
When adding (for stunnel) 101.start and 101.stop to the /etc/vz/conf folder and restarting the container, it locks up and we cannot stop it. It appears that the LSB tags from /etc/init.d/stunnel4 file need to be removed first when insserv is used.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back