ProxMox 4.1: Backup Fails if the Container is running

[mbagni]

Hi,

Since version 4.0 backing up a running container or a VM was working fine. After the upgrade to 4.1 if the container is running and the backup has the "stop" option (stop the CT, make the backup and restart) it seems not to wait for the running container to stop and when it mounts the container's raw file under /mnt/vzsnap0 it fails with the following error:

INFO: stopping vm
INFO: creating archive '/media/vms/dump/vzdump-lxc-400-2015_12_19-14_08_12.tar.lzo'
mount: wrong fs type, bad option, bad superblock on /dev/loop7,
missing codepage or helper program, or other error

In some cases useful info is found in syslog - try
dmesg | tail or so.
umount: /mnt/vzsnap0/: not mounted
command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32

In 2 out of 7 containers running, the backup worked fine but it clearly took much more time to shutdown the running CT with respect the faulty ones (the successful backups took 50-60 seconds between the log message "INFO: stopping vm" and the next message "INFO: creating archive ...", while for the faulty one that time lapse lasted 15-25 seconds).

If the container is switched off manually before doing the backup, it works fine.

Thanks in advance!

 

[RobFantini]

same here, see bug https://bugzilla.proxmox.com/show_bug.cgi?id=771 .

If it is the same issue can you reply and add to that bug report?

 

[pizza]

I've noticed the same problem. Backup failed on containers created with 4.1. Containers created in v4.0 can still backup

 

[dietmar]

What storage type do you use for the container?

 

[Ivan68]

I've noticed the same problem. Backup failed on containers created with 4.1. Containers created in v4.0 can still backup

When I use my local storage backup is ok. With NFS storage I have the same problem.

 

[mbagni]

Hi Dietmar,
if your question was addressed to me, here is the answer:
The Containers are hosted on a Ext4 file system mounted on LVM and physically they are hosted on a SSD. Inside the Container I have Linux Debian 8 and 7 (the problem is common on all CTs).
The backup repository is a couple of 500 GB HD used in mirror (SW raid using Linux MD) and managed under LVM in one unique partition of 500 GB.
The processor is an Intel Core i7-5960X @ 3.00GHz (it is not an issue of power!).

 

[dietmar]

Please can you test with latest version from pvetest? Assuming all other packages are up to date, you can do:

# wget http://download1.proxmox.com/debian/dists/jessie/pvetest/binary-amd64/pve-container_1.0-36_all.deb
# dpkg -i pve-container_1.0-36_all.deb

 

[krli]

Hello!
Same problem here. Scheduled backup of an LXC container always fails with the next error. Manual backup after stopping the container works fine.

Local storage: ext4 over LVM.
Backups storage: NFS
$ pveversion --> pve-manager/4.1-1/2f9650d4 (running kernel: 4.2.6-1-pve)

100: dic 19 23:59:06 INFO: Starting Backup of VM 100 (lxc)
100: dic 19 23:59:06 INFO: status = running
100: dic 19 23:59:06 INFO: backup mode: stop
100: dic 19 23:59:06 INFO: ionice priority: 7
100: dic 19 23:59:06 INFO: stopping vm
100: dic 19 23:59:12 INFO: creating archive '/mnt/pve/Backups/dump/vzdump-lxc-100-2015_12_19-23_59_02.tar.lzo'
100: dic 19 23:59:33 INFO: restarting vm
100: dic 19 23:59:34 INFO: vm is online again after 28 seconds
100: dic 19 23:59:34 ERROR: Backup of VM 100 failed - command 'mount -o loop /vz/images/100/vm-100-disk-1.raw /mnt/vzsnap0/' failed: exit code 32

I think DMESG shows some warnings that may be related:

[898940.529963] EXT4-fs warning (device loop1): ext4_multi_mount_protect:318: MMP interval 42 higher than expected, please wait.

[898961.520852] EXT4-fs warning (device loop1): ext4_multi_mount_protect:331: Device is already active on another node.
[898961.520855] EXT4-fs warning (device loop1): ext4_multi_mount_protect:331: MMP failure info: last update time: 1450738754, last update node: HServer01, last update device: loop0

.....

[898962.904589] audit: type=1400 audit(1450738776.085:89): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/run/" pid=57425 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
[898962.907384] audit: type=1400 audit(1450738776.085:90): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/run/lock/" pid=57436 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[898962.909481] audit: type=1400 audit(1450738776.089:91): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/proc/" pid=57443 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[898962.912084] audit: type=1400 audit(1450738776.089:92): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/sys/" pid=57451 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
[898962.929536] audit: type=1400 audit(1450738776.109:93): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=57518 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[898962.932064] audit: type=1400 audit(1450738776.109:94): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/dev/pts/" pid=57526 comm="mount" flags="rw, nosuid, noexec, remount, relatime"

Looking around I found adding the next line to
/etc/apparmor.d/lxc/lxc-default could help, but it doesn't change anything.

mount options=(rw, bind, ro, remount, relatime),

I hope my information helps.

Thanks,

 

[mbagni]

Updated today to the latest release of the kernel as suggested by the system (Kernel 4.2.6.1) installed pve-container_1.0-36_all.deb and tested on a container that used to give error: IT WORKS!

Thanks!