LXC backup doesn't finish

[voarsh]

I have a backup job that's been trying to run for over 15 hours.

Backups for this LXC has been killed manually for over a week now because it doesn't finish.

There's no local location (with enough space) except NFS drives. The LXC is only 30-38 GB's in size and the same NFS drives work for other VM's and LXC's that are bigger (300 GB's etc).

Anything I can do to work out the problem?

LXC config:

arch: amd64
cores: 16
cpulimit: 16
features: nesting=1
hostname: os
lock: backup
memory: 4224
mp0: /mnt/bindmounts/shared2,mp=/shared
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.100.1,hwaddr=5E:23:43:5C:9D:8F,ip=192.168.100.101/24,type=veth
onboot: 1
ostype: ubuntu
parent: vzdump
rootfs: HPBay8:vm-113-disk-0,size=38G
swap: 1536

[vzdump]
#vzdump backup snapshot
arch: amd64
cores: 4
cpulimit: 4
features: nesting=1
hostname:os
memory: 3616
mp0: /mnt/bindmounts/shared,mp=/shared
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=5E:23:43:5C:9D:8F,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-113-disk-0,size=34G
snaptime: 1609347146
swap: 1536

Backup log:

INFO: trying to get global lock - waiting...
INFO: got global lock
INFO: starting new backup job: vzdump 113 --node HPProliantDL360PGen8 --remove 0 --compress zstd --mode snapshot --storage FourTBpveIPC2Expansion
INFO: Starting Backup of VM 113 (lxc)
INFO: Backup started at 2021-01-16 05:27:20
INFO: status = running
INFO: CT Name: os
INFO: excluding bind mount point mp0 ('/shared') from backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: homelabos
INFO: excluding bind mount point mp0 ('/shared') from backup
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/23073/root// to /mnt/pve/FourTBpveIPC2Expansion/dump/vzdump-lxc-113-2021_01_16-05_26_47.tmp

FYI, that snapshot doesn't seem to exist, especially where it is looking (the local), I was thinking this might be a problem?

- edit - I removed the missing snapshot and it still doesn't complete the backup

 

[voarsh]

Update:
Powering off seems to have the LXC backup complete in 30 mins:

INFO: starting new backup job: vzdump 113 --remove 0 --compress lzo --mode snapshot --node HPProliantDL360PGen8 --storage FourteenTBExpansionUSB4

INFO: Starting Backup of VM 113 (lxc)

INFO: Backup started at 2021-01-17 04:26:17

INFO: status = stopped

INFO: backup mode: stop

INFO: ionice priority: 7

INFO: CT Name: os

INFO: excluding bind mount point mp0 ('/shared') from backup

INFO: creating archive '/mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-04_26_13.tar.lzo'

INFO: Total bytes written: 30359183360 (29GiB, 18MiB/s)

INFO: archive file size: 15.67GB

INFO: Finished Backup of VM 113 (00:27:55)

INFO: Backup finished at 2021-01-17 04:54:08

INFO: Backup job finished successfully

TASK OK

I tried again earlier with it on and got some strange results:

INFO: starting new backup job: vzdump 113 --storage FourteenTBExpansionUSB4 --node HPProliantDL360PGen8 --mode snapshot --remove 0 --compress lzo
INFO: Starting Backup of VM 113 (lxc)
INFO: Backup started at 2021-01-17 01:57:52
INFO: status = running
INFO: CT Name: os
INFO: excluding bind mount point mp0 ('/shared') from backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: os
INFO: excluding bind mount point mp0 ('/shared') from backup
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/2183/root// to /mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-01_57_52.tmp
INFO: rsync: mknod "/mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-01_57_52.tmp/var/lib/docker/overlay2/051e3378c46598f321d7ae7129f52c50c44a6f5ff0579b4606880a01b23dcf41/diff/etc/rc2.d/S01ondemand" failed: Operation not permitted (1)
INFO: rsync: mknod "/mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-01_57_52.tmp/var/lib/docker/overlay2/051e3378c46598f321d7ae7129f52c50c44a6f5ff0579b4606880a01b23dcf41/diff/etc/rc2.d/S01rc.local" failed: Operation not permitted (1)
INFO: rsync: mknod "/mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-01_57_52.tmp/var/lib/docker/overlay2/051e3378c46598f321d7ae7129f52c50c44a6f5ff0579b4606880a01b23dcf41/diff/etc/rc3.d/S01ondemand" failed: Operation not permitted (1)
INFO: rsync: mknod "/mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-01_57_52.tmp/var/lib/docker/overlay2/051e3378c46598f321d7ae7129f52c50c44a6f5ff0579b4606880a01b23dcf41/diff/etc/rc3.d/S01rc.local" failed: Operation not permitted (1)
INFO: rsync: mknod "/mnt/pve/FourteenTBExpansionUSB4/dump/vzdump-lxc-113-2021_01_17-01_57_52.tmp/var/lib/docker/overlay2/051e3378c46598f321d7ae7129f52c50c44a6f5ff0579b4606880a01b23dcf41/diff/etc/rc4.d/S01ondemand" failed: Operation not permitted (1)

 

[fiona]

Hi,
could you tell us your storage configuration /etc/pve/storage.cfg?

It seems like the storage the container is on does not support snapshots, so the backup defaults to suspend mode. But suspend mode needs to copy all data twice which is especially bad for performance if your temporary directory is also non-local (if you do have enough space try configuring it to a local directory.

Regarding the mknod errors, please share the output of pveversion -v.