ERROR: Backup of VM failed ... exit code 32

Discussion in 'Proxmox VE: Installation and configuration' started by mlanner, Aug 29, 2018.

  1. mlanner

    mlanner Member

    Joined:
    Apr 1, 2009
    Messages:
    184
    Likes Received:
    1
    I have an LXC container running on a Ceph pool. It's running just fine. It has a nightly backup set up to run. For the last week or so, the backup keeps on failing with:

    Code:
    vzdump 60180 --mailnotification always --mode snapshot --mailto someone@example.com --quiet 1 --storage nas01-backup-7x --compress lzo
    
    60180: 2018-08-29 01:00:02 INFO: Starting Backup of VM 60180 (lxc)
    60180: 2018-08-29 01:00:02 INFO: status = running
    60180: 2018-08-29 01:00:02 INFO: CT Name: my-lxc-container
    60180: 2018-08-29 01:00:02 INFO: found old vzdump snapshot (force removal)
    60180: 2018-08-29 01:00:02 INFO: backup mode: snapshot
    60180: 2018-08-29 01:00:02 INFO: ionice priority: 7
    60180: 2018-08-29 01:00:02 INFO: create storage snapshot 'vzdump'
    60180: 2018-08-29 01:00:03 ERROR: Backup of VM 60180 failed - command 'mount -o ro,noload /dev/rbd2 /mnt/vzsnap0//' failed: exit code 32
    
    If I list snapshots of that container, I can see a snapshot left behind.

    Code:
    # rbd --pool ceph snap ls vm-60180-disk-1
    SNAPID NAME       SIZE TIMESTAMP
        63 vzdump 65536 MB Wed Aug 29 01:00:03 2018
    
    Even if I clean up the snapshot with:

    Code:
    # rbd snap rm ceph/vm-60180-disk-1@vzdump
    
    ... it doesn't help.

    If I run a manual backup to the same storage backend, a NAS, but a different mount point, it generally works. I'm a bit baffled as to why this is failing.

    Anyone have any suggestions or ideas for why this is failing?
     
  2. GPLExpert

    GPLExpert New Member
    Proxmox Subscriber

    Joined:
    Jul 18, 2016
    Messages:
    19
    Likes Received:
    0
    Hello,

    I have the same problem an no solution too since 2 weeks with one container.
     
  3. mlanner

    mlanner Member

    Joined:
    Apr 1, 2009
    Messages:
    184
    Likes Received:
    1
    BUMP ... No one has a solution for this? If I shut down the container and back it up manually, I can get a backup to work. However, that's really not a great way to go about it.
     
  4. Jonathan Thompson

    Jonathan Thompson New Member

    Joined:
    Dec 30, 2018
    Messages:
    2
    Likes Received:
    0
    BUMP! I'm getting this too. LXC container on Ceph storage.

    Code:
    INFO: starting new backup job: vzdump 101 --node vmhost12 --remove 0 --compress lzo --storage local --mode snapshot
    INFO: Starting Backup of VM 101 (lxc)
    INFO: status = running
    INFO: CT Name: Unifi-Controller
    INFO: found old vzdump snapshot (force removal)
    rbd: sysfs write failed
    can't unmap rbd device /dev/rbd/Share/vm-101-disk-0@vzdump: rbd: sysfs write failed
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: create storage snapshot 'vzdump'
    mount: /dev/rbd4 is already mounted or /mnt/vzsnap0 busy
    umount: /mnt/vzsnap0/: not mounted
    command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
    ERROR: Backup of VM 101 failed - command 'mount -o ro,noload /dev/rbd4 /mnt/vzsnap0//' failed: exit code 32
    INFO: Backup job finished with errors
    TASK ERROR: job errors
     
  5. SlowMo

    SlowMo New Member
    Proxmox Subscriber

    Joined:
    Jan 16, 2018
    Messages:
    2
    Likes Received:
    0
    Hello. Since some days i have the same Problem. I cleaned up everything and try the Backup on a local and remote disc but still not working.
    The Container run's on a Ceph-Storage

    Local Backup-Point (ZFS)
    Code:
    INFO: starting new backup job: vzdump 116 --mode snapshot --remove 0 --compress lzo --node xm-f-pv01 --storage local
    INFO: filesystem type on dumpdir is 'zfs' -using /var/tmp/vzdumptmp2009437 for temporary files
    INFO: Starting Backup of VM 116 (lxc)
    INFO: status = running
    INFO: CT Name: jira
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: create storage snapshot 'vzdump'
    mount: /dev/rbd11 is already mounted or /mnt/vzsnap0 busy
    umount: /mnt/vzsnap0/: not mounted
    command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
    ERROR: Backup of VM 116 failed - command 'mount -o ro,noload /dev/rbd11 /mnt/vzsnap0//' failed: exit code 32
    INFO: Backup job finished with errors
    TASK ERROR: job errors
    
    Remote Backup-Point (CIFS)
    Code:
    INFO: starting new backup job: vzdump 116 --storage mst_nas01_bak_man --mode snapshot --node xm-f-pv01 --compress lzo --remove 0
    INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp2032104 for temporary files
    INFO: Starting Backup of VM 116 (lxc)
    INFO: status = running
    INFO: CT Name: jira
    INFO: found old vzdump snapshot (force removal)
    rbd: sysfs write failed
    can't unmap rbd volume vm-116-disk-1: rbd: sysfs write failed
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: create storage snapshot 'vzdump'
    snapshot create failed: starting cleanup
    no lock found trying to remove 'backup' lock
    ERROR: Backup of VM 116 failed - rbd snapshot 'vm-116-disk-1' error: rbd: failed to create snapshot: (17) File exists
    INFO: Backup job finished with errors
    TASK ERROR: job errors
    
    My PVEVERSION
    Code:
    pveversion -v
    proxmox-ve: 5.2-2 (running kernel: 4.15.18-4-pve)
    pve-manager: 5.2-9 (running version: 5.2-9/4b30e8f9)
    pve-kernel-4.15: 5.2-7
    pve-kernel-4.15.18-4-pve: 4.15.18-23
    pve-kernel-4.15.18-1-pve: 4.15.18-19
    pve-kernel-4.15.17-1-pve: 4.15.17-9
    pve-kernel-4.13.13-6-pve: 4.13.13-42
    pve-kernel-4.13.13-5-pve: 4.13.13-38
    pve-kernel-4.13.13-2-pve: 4.13.13-33
    ceph: 12.2.8-pve1
    corosync: 2.4.2-pve5
    criu: 2.11.1-1~bpo90
    glusterfs-client: 3.8.8-1
    ksm-control-daemon: 1.2-2
    libjs-extjs: 6.0.1-2
    libpve-access-control: 5.0-8
    libpve-apiclient-perl: 2.0-5
    libpve-common-perl: 5.0-38
    libpve-guest-common-perl: 2.0-18
    libpve-http-server-perl: 2.0-10
    libpve-storage-perl: 5.0-29
    libqb0: 1.0.1-1
    lvm2: 2.02.168-pve6
    lxc-pve: 3.0.2+pve1-2
    lxcfs: 3.0.2-2
    novnc-pve: 1.0.0-2
    proxmox-widget-toolkit: 1.0-20
    pve-cluster: 5.0-30
    pve-container: 2.0-27
    pve-docs: 5.2-8
    pve-firewall: 3.0-14
    pve-firmware: 2.0-5
    pve-ha-manager: 2.0-5
    pve-i18n: 1.0-6
    pve-libspice-server1: 0.12.8-3
    pve-qemu-kvm: 2.11.2-1
    pve-xtermjs: 1.0-5
    qemu-server: 5.0-35
    smartmontools: 6.5+svn4324-1
    spiceterm: 3.0-5
    vncterm: 1.5-3
    zfsutils-linux: 0.7.9-pve1~bpo9
    
     
  6. tom

    tom Proxmox Staff Member
    Staff Member

    Joined:
    Aug 29, 2006
    Messages:
    13,551
    Likes Received:
    405
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. ats08

    ats08 New Member

    Joined:
    Feb 18, 2018
    Messages:
    1
    Likes Received:
    0
    Same problem here. LXC on Ceph, backing up to CIFS. Strangely, I have several containers, but only one (debian 9 running a Unifi controller) is having the issue.

    Code:
    
    Task viewer: VM/CT 108 - Backup
    
    OutputStatus
    
    Stop
    INFO: starting new backup job: vzdump 108 --compress lzo --remove 0 --storage cluster --mode snapshot --node pve2
    INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3290365 for temporary files
    INFO: Starting Backup of VM 108 (lxc)
    INFO: status = running
    INFO: CT Name: unifi
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: create storage snapshot 'vzdump'
    mount: /dev/rbd6 is already mounted or /mnt/vzsnap0 busy
    umount: /mnt/vzsnap0/: not mounted
    command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
    ERROR: Backup of VM 108 failed - command 'mount -o ro,noload /dev/rbd6 /mnt/vzsnap0//' failed: exit code 32
    INFO: Backup job finished with errors
    TASK ERROR: job errors
    
     
  8. DonatasJa

    DonatasJa New Member
    Proxmox Subscriber

    Joined:
    Sep 18, 2017
    Messages:
    2
    Likes Received:
    0
    I also have the same problem, backup sometimes fail.
    It seems there are because ceph snapshot are not atomic.
    vzdump can't mount ro snapshot, because need to replay journal log, but on ro mount it's not possible.
    Maybe anybody have a solution?


    May 10 07:47:46 e21 vzdump[640890]: <root@pam> starting task UPID:e21:0009C782:002DC8EA:5CD50272:vzdump:1115:root@pam:
    May 10 07:47:46 e21 kernel: [30005.937204] rbd: rbd12: capacity 161061273600 features 0x1
    May 10 07:47:46 e21 kernel: [30005.981517] EXT4-fs (rbd12): mounted filesystem without journal. Opts: noload
    May 10 07:52:03 e21 kernel: [30262.680810] ------------[ cut here ]------------
    May 10 07:52:03 e21 kernel: [30262.686296] invalid opcode: 0000 [#1] SMP PTI
     
  9. SlowMo

    SlowMo New Member
    Proxmox Subscriber

    Joined:
    Jan 16, 2018
    Messages:
    2
    Likes Received:
    0
    Hi Tom. I updated to a newer Version but the problem still occurs again after some time. It only suxxx on Lxc-Containers.

    Code:
    NFO: starting new backup job: vzdump 116 113 117 144 174 118 121 --mailnotification always --compress lzo --mailto server@***** --mode snapshot --quiet 1 --storage xm_mst_nas01_bak_7 --node xm-f-pv01
    INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3312603 for temporary files
    INFO: Starting Backup of VM 113 (lxc)
    INFO: status = running
    INFO: CT Name: xm-mst-wiki-live16
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: create storage snapshot 'vzdump'
    mount: /dev/rbd0 is already mounted or /mnt/vzsnap0 busy
    umount: /mnt/vzsnap0/: not mounted
    command 'umount -l -d /mnt/vzsnap0/' failed: exit code 32
    ERROR: Backup of VM 113 failed - command 'mount -o ro,noload /dev/rbd0 /mnt/vzsnap0//' failed: exit code 32
    INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp3312603 for temporary files
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice