LXC backup failed: exit code 23

[ichihaifu]

I started having this problem during backup of one of my containers, since upgrading to 6.2 from 5.4 last month:

INFO: starting new backup job: vzdump 110 --storage storage_backup --mode snapshot --remove 0 --compress zstd --node pve
INFO: Starting Backup of VM 110 (lxc)
INFO: Backup started at 2020-08-19 10:49:26
INFO: status = running
INFO: CT Name: --
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/database') in backup
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: --
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/database') in backup
INFO: starting first sync /proc/9038/root/ to /zpool/temp/vzdumptmp14972
ERROR: Backup of VM 110 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/9038/root//./ /proc/9038/root//./database /zpool/temp/vzdumptmp14972' failed: exit code 23
INFO: Failed at 2020-08-19 11:00:23
INFO: Backup job finished with errors
TASK ERROR: job errors

Spoiler: Config files

Container config:

# cat /etc/pve/lxc/110.conf
arch: amd64
cores: 4
hostname: --
memory: 8192
mp0: storage_databases:110/vm-110-disk-0.raw,mp=/database,backup=1,mountoptions=noatime,size=50G
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=3A:C7:CB:D8:1C:5D,type=veth
onboot: 1
ostype: centos
rootfs: ssdpool:subvol-110-disk-0,size=6G
startup: order=3
swap: 0
unprivileged: 1

Storage config:

# cat /etc/pve/storage.cfg
..
nfs: storage_databases
        export --
        path /mnt/pve/storage_databases
        server 192.168.255.1
        content rootdir
        options vers=4.1

All other container backups on the same task work fine, although also no other container has resources on a network storage.
I found this thread with same error (LXC backup fails with "failed: exit code 23"), but I seem to have a little bit different symptoms (I tried adjusting ACL's anyway and that had no effect).
Using Stop backup mode works, but I want to avoid using this option.

The container is running a MySQL database, if that makes any difference.

Did something change between major versions that would cause this issue?

 

[oguz]

hi,

i see storage_databases is nfs.

just to test, can you mark the database mp as backup=0 and try again? this way we know if it's related to the mountpoint or the container itself.

 

[ichihaifu]

With backup=0 flag on the database mount, backup in snapshot mode works, however, suspend mode does not (see output below). I guess that would explain why other container backups succeed, as they run in snapshot mode.

Suspend:

INFO: starting new backup job: vzdump 110 --compress zstd --remove 0 --node pve --storage storage_backup --mode suspend
INFO: Starting Backup of VM 110 (lxc)
INFO: Backup started at 2020-08-19 20:14:34
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: --
INFO: including mount point rootfs ('/') in backup
INFO: excluding volume mount point mp0 ('/database') from backup (disabled)
INFO: starting first sync /proc/6864/root/ to /zpool/temp/vzdumptmp15837
ERROR: Backup of VM 110 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' '--exclude=/database' /proc/6864/root//./ /zpool/temp/vzdumptmp15837' failed: exit code 23
INFO: Failed at 2020-08-19 20:14:55
INFO: Backup job finished with errors
TASK ERROR: job errors

Spoiler: Snapshot

INFO: starting new backup job: vzdump 110 --compress zstd --remove 0 --node pve --storage storage_backup --mode snapshot
INFO: Starting Backup of VM 110 (lxc)
INFO: Backup started at 2020-08-19 20:14:03
INFO: status = running
INFO: CT Name: --
INFO: including mount point rootfs ('/') in backup
INFO: excluding volume mount point mp0 ('/database') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating vzdump archive '/mnt/pve/storage_backup/dump/vzdump-lxc-110-2020_08_19-20_14_03.tar.zst'
INFO: tar: 0f529d12b9e874ca8662b1cc5fb2de0d05fd8dd93d6663abd98758b9fb7f1a38-primary.xml.gz: Warning: Cannot flistxattr: Argument list too long
INFO: tar: 864ad0d0d191bd8b04048d31eaf6996c05661d1abed8a767db0a937d0767a9f4-primary.xml.gz: Warning: Cannot flistxattr: Argument list too long
INFO: Total bytes written: 988456960 (943MiB, 77MiB/s)
INFO: archive file size: 286MB
INFO: remove vzdump snapshot
INFO: Finished Backup of VM 110 (00:00:16)
INFO: Backup finished at 2020-08-19 20:14:19
INFO: Backup job finished successfully
TASK OK