ERROR: Backup of VM failed ... exit code 32

[mlanner]

I have an LXC container running on a Ceph pool. It's running just fine. It has a nightly backup set up to run. For the last week or so, the backup keeps on failing with:

vzdump 60180 --mailnotification always --mode snapshot --mailto someone@example.com --quiet 1 --storage nas01-backup-7x --compress lzo

60180: 2018-08-29 01:00:02 INFO: Starting Backup of VM 60180 (lxc)
60180: 2018-08-29 01:00:02 INFO: status = running
60180: 2018-08-29 01:00:02 INFO: CT Name: my-lxc-container
60180: 2018-08-29 01:00:02 INFO: found old vzdump snapshot (force removal)
60180: 2018-08-29 01:00:02 INFO: backup mode: snapshot
60180: 2018-08-29 01:00:02 INFO: ionice priority: 7
60180: 2018-08-29 01:00:02 INFO: create storage snapshot 'vzdump'
60180: 2018-08-29 01:00:03 ERROR: Backup of VM 60180 failed - command 'mount -o ro,noload /dev/rbd2 /mnt/vzsnap0//' failed: exit code 32

If I list snapshots of that container, I can see a snapshot left behind.

# rbd --pool ceph snap ls vm-60180-disk-1
SNAPID NAME       SIZE TIMESTAMP
    63 vzdump 65536 MB Wed Aug 29 01:00:03 2018

Even if I clean up the snapshot with:

# rbd snap rm ceph/vm-60180-disk-1@vzdump

... it doesn't help.

If I run a manual backup to the same storage backend, a NAS, but a different mount point, it generally works. I'm a bit baffled as to why this is failing.

Anyone have any suggestions or ideas for why this is failing?

 

[GPLExpert]

Hello,

I have the same problem an no solution too since 2 weeks with one container.

 

[mlanner]

BUMP ... No one has a solution for this? If I shut down the container and back it up manually, I can get a backup to work. However, that's really not a great way to go about it.

 

[Jonathan Thompson]

BUMP! I'm getting this too. LXC container on Ceph storage.

INFO: starting new backup job: vzdump 101 --node vmhost12 --remove 0 --compress lzo --storage local --mode snapshot
INFO: Starting Backup of VM 101 (lxc)
INFO: status = running
INFO: CT Name: Unifi-Controller
INFO: found old vzdump snapshot (force removal)
rbd: sysfs write failed
can't unmap rbd device /dev/rbd/Share/vm-101-disk-0@vzdump: rbd: sysfs write failed
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
mount: /dev/rbd4 is already mounted or /mnt/vzsnap0 busy
umount: /mnt/vzsnap0/: not mounted
command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
ERROR: Backup of VM 101 failed - command 'mount -o ro,noload /dev/rbd4 /mnt/vzsnap0//' failed: exit code 32
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[SlowMo]

Hello. Since some days i have the same Problem. I cleaned up everything and try the Backup on a local and remote disc but still not working.
The Container run's on a Ceph-Storage

Local Backup-Point (ZFS)

INFO: starting new backup job: vzdump 116 --mode snapshot --remove 0 --compress lzo --node xm-f-pv01 --storage local
INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp2009437 for temporary files
INFO: Starting Backup of VM 116 (lxc)
INFO: status = running
INFO: CT Name: jira
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
mount: /dev/rbd11 is already mounted or /mnt/vzsnap0 busy
umount: /mnt/vzsnap0/: not mounted
command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
ERROR: Backup of VM 116 failed - command 'mount -o ro,noload /dev/rbd11 /mnt/vzsnap0//' failed: exit code 32
INFO: Backup job finished with errors
TASK ERROR: job errors

Remote Backup-Point (CIFS)

INFO: starting new backup job: vzdump 116 --storage mst_nas01_bak_man --mode snapshot --node xm-f-pv01 --compress lzo --remove 0
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp2032104 for temporary files
INFO: Starting Backup of VM 116 (lxc)
INFO: status = running
INFO: CT Name: jira
INFO: found old vzdump snapshot (force removal)
rbd: sysfs write failed
can't unmap rbd volume vm-116-disk-1: rbd: sysfs write failed
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
snapshot create failed: starting cleanup
no lock found trying to remove 'backup' lock
ERROR: Backup of VM 116 failed - rbd snapshot 'vm-116-disk-1' error: rbd: failed to create snapshot: (17) File exists
INFO: Backup job finished with errors
TASK ERROR: job errors

My PVEVERSION

pveversion -v
proxmox-ve: 5.2-2 (running kernel: 4.15.18-4-pve)
pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
pve-kernel-4.15: 5.2-7
pve-kernel-4.15.18-4-pve: 4.15.18-23
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
pve-kernel-4.13.13-6-pve: 4.13.13-42
pve-kernel-4.13.13-5-pve: 4.13.13-38
pve-kernel-4.13.13-2-pve: 4.13.13-33
ceph: 12.2.8-pve1
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-38
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-10
libpve-storage-perl: 5.0-29
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-2
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-20
pve-cluster: 5.0-30
pve-container: 2.0-27
pve-docs: 5.2-8
pve-firewall: 3.0-14
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.2-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-35
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

 

[tom]

My PVEVERSION

You run a quit old packages, please upgrade to current version and test again.

https://pve.proxmox.com/wiki/Downlo...Proxmox_Virtual_Environment_5.x_to_latest_5.3

 

[ats08]

Same problem here. LXC on Ceph, backing up to CIFS. Strangely, I have several containers, but only one (debian 9 running a Unifi controller) is having the issue.

Task viewer: VM/CT 108 - Backup

OutputStatus

Stop
INFO: starting new backup job: vzdump 108 --compress lzo --remove 0 --storage cluster --mode snapshot --node pve2
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3290365 for temporary files
INFO: Starting Backup of VM 108 (lxc)
INFO: status = running
INFO: CT Name: unifi
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
mount: /dev/rbd6 is already mounted or /mnt/vzsnap0 busy
umount: /mnt/vzsnap0/: not mounted
command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
ERROR: Backup of VM 108 failed - command 'mount -o ro,noload /dev/rbd6 /mnt/vzsnap0//' failed: exit code 32
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[DonatasJa]

I also have the same problem, backup sometimes fail.
It seems there are because ceph snapshot are not atomic.
vzdump can't mount ro snapshot, because need to replay journal log, but on ro mount it's not possible.
Maybe anybody have a solution?


May 10 07:47:46 e21 vzdump[640890]: <root@pam> starting task UPID:e21:0009C782:002DC8EA:5CD50272:vzdump:1115:root@pam:
May 10 07:47:46 e21 kernel: [30005.937204] rbd: rbd12: capacity 161061273600 features 0x1
May 10 07:47:46 e21 kernel: [30005.981517] EXT4-fs (rbd12): mounted filesystem without journal. Opts: noload
May 10 07:52:03 e21 kernel: [30262.680810] ------------[ cut here ]------------
May 10 07:52:03 e21 kernel: [30262.686296] invalid opcode: 0000 [#1] SMP PTI

 

[SlowMo]

You run a quit old packages, please upgrade to current version and test again.

https://pve.proxmox.com/wiki/Downlo...Proxmox_Virtual_Environment_5.x_to_latest_5.3

Hi Tom. I updated to a newer Version but the problem still occurs again after some time. It only suxxx on Lxc-Containers.

NFO: starting new backup job: vzdump 116 113 117 144 174 118 121 --mailnotification always --compress lzo --mailto server@***** --mode snapshot --quiet 1 --storage xm_mst_nas01_bak_7 --node xm-f-pv01
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3312603 for temporary files
INFO: Starting Backup of VM 113 (lxc)
INFO: status = running
INFO: CT Name: xm-mst-wiki-live16
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
mount: /dev/rbd0 is already mounted or /mnt/vzsnap0 busy
umount: /mnt/vzsnap0/: not mounted
command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
ERROR: Backup of VM 113 failed - command 'mount -o ro,noload /dev/rbd0 /mnt/vzsnap0//' failed: exit code 32
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3312603 for temporary files

 

[ednt]

In our case there are daemonized tar jobs which blocks the device.
You can use

rbd showmapped

to see the used devices.
And

fuser -amv /dev/rbdX

to see who is using this device.
But since in our case it were already daemonized tar jobs, the only solution was a reboot.