[SOLVED] LXC Backup faild ( exit code 23 )

demon_mono · Nov 24, 2020

Hello,

One of my CT can not be saved since few days. I don't explain why . Here is the save log :

Code:

INFO: starting new backup job: vzdump 105 --compress zstd --storage save_vz --node server --mode suspend --remove 0
INFO: Starting Backup of VM 105 (lxc)
INFO: Backup started at 2020-11-24 17:09:48
INFO: status = running
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: devloprog-nextcloud
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/www/nextcloud/data/') in backup
INFO: starting first sync /proc/2514103/root/ to /SAVE_VZ/dump/vzdump-lxc-105-2020_11_24-17_09_48.tmp
ERROR: Backup of VM 105 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --sparse --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/2514103/root//./ /proc/2514103/root//./var/www/nextcloud/data/ /SAVE_VZ/dump/vzdump-lxc-105-2020_11_24-17_09_48.tmp' failed: exit code 23
INFO: Failed at 2020-11-24 17:17:36
INFO: Backup job finished with errors
TASK ERROR: job errors

Regards,

fabian · Nov 25, 2020

rsync returning 23 means 'Partial transfer due to error'. does a snapshot (if all your volumes support it) or stop mode backup work? alternatively, you could try adding '-v --progress' to the rsync command-line arguments, change the target directory, and attempt the rsync manually..

demon_mono · Nov 25, 2020

fabian said:
-v --progress

I am trying this , I will tell you what it retuns.

demon_mono · Nov 25, 2020

One file could not be saved due to read error. I delete it , and that works !

I keep in mind the rsync verbose option.

Thanks !

AnK · Feb 6, 2022

Hi,

I am new to Proxmox and special to lxc.
I have my storage configured the oldschool way, that means LVM, ext4 and /var/lib/vz.
When I try to take a backup of a lxc container I get the error mentioned above.
Would be LVM thin a solution for me, or do I have to deal with zfs, if I would be able to take snapshots?

fabian · Feb 7, 2022

both LVM thin and ZFS support snapshots - see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_storage

AnK · Feb 7, 2022

Hi fabian,

thanks for your answer!
In the meantime I have tried both, with Docker inside LXC (I know it's not the prefered/supported way) and both are running fine.

For ZFS I have to enable fuse-overlayfs storage driver in Docker:
https://c-goes.github.io/posts/proxmox-lxc-docker-fuse-overlayfs/

Therefore I think I am going with lvm-thin for performing a snaphost backup on the lxc containers.

amorphis · Sep 8, 2022

Hi All!
Have the same issue. One of my LXC failed to backup:
NFO: resume vm
INFO: guest is online again after 1 seconds
ERROR: Backup of VM 110 failed - command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/8865/root//./ /vms/temp/vzdumptmp3583133_110/' failed: exit code 23
INFO: Failed at 2022-09-08 05:22:56
INFO: Backup job finished with errors
TASK ERROR: job errors

When I try to run command from console(rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/8865/root//./ /vms/temp/vzdumptmp3583133_110/') - I have no error.

Is a way to change rsync options for GUI/backup job?

I have no idea. Please help!

fabian · Sep 8, 2022

you have to find out the reason - there are several which are possible (PHP session files or similar directories with unwritable-by-root permissions in Debian-based containers, read errors, ACL problems, ..)

amorphis · Sep 9, 2022

Thank you fabian!
I try to find out a reason by adding -v --progress to /usr/share/perl5/PVE/VZDump/LXC.pm
but seems changes not applied.

my $rsync = ['rsync', '-v', '--progress', '--stats', '-h', @xattr, '--numeric-ids',
'-aH', '--delete', '--no-whole-file',
($first ? '--sparse' : '--inplace'),
'--one-file-system', '--relative'];

Output still
command 'rsync --stats -h -X -A --numeric-ids -aH --delete --no-whole-file --inplace --one-file-system --relative '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' /proc/8865/root//./ /vms/temp/vzdumptmp1157870_110/' failed: exit code 23

I'll try to apply patch described in https://lists.proxmox.com/pipermail/pve-devel/2021-September/049894.html
bun not sure how to apply it correctly.

amorphis · Sep 9, 2022

Found problem. One of dirs had sticky bit set.
Thank you!

MaFL · Oct 17, 2022

fabian said:
you have to find out the reason - there are several which are possible (PHP session files or similar directories with unwritable-by-root permissions in Debian-based containers, read errors, ACL problems, ..)

Hi Fabian,
is there any reason why your fix is not commited?

fabian · Oct 18, 2022

I pinged it a few weeks ago

mangelot · Nov 9, 2022

@fabian

Is it possible to add "-av --progress" to the default /usr/share/perl5/PVE/VZDump/LXC.pm script or is there any other way to achieve the same output?
So all LXC backups have this option enabled by default? or add a checkbox "show progress/debug" to the backup task in Proxmox GUI?

Best regards,

Marco

fabian · Nov 9, 2022

mangelot said:
@fabian

Is it possible to add "-av --progress" to the default /usr/share/perl5/PVE/VZDump/LXC.pm script or is there any other way to achieve the same output?
So all LXC backups have this option enabled by default? or add a checkbox "show progress/debug" to the backup task in Proxmox GUI?

Best regards,

Marco

that would blow up the task log for no reason.. you can always manually add that in (and restart/reload PVE services afterwards) when debugging though. note that the patch in question (dropping in place overwriting of changed files) that fixes some of the common occurrences of error 23 is now applied in git and will be released with the next pve-container version!

Zubin Singh Parihar · Dec 30, 2023

The typical problem is that you are running ZFS without POSIX ACL Support.

The LXC container has ACL settings inside its filesystem and the 'snaphot' backup process that the Proxmox VE host runs is an rsync to the /var/tmp directory. If POSIX ACL is not turned on in the rpool/ROOT/pve-1 dataset (and it isn't by default for whatever strange reason, and the Proxmox devs should and hopefully will do that in the next release), then the rsync will fail.

TEST:

Bash:

$ zfs get acltype rpool/ROOT/pve-1

if it returns:

Bash:

NAME              PROPERTY  VALUE     SOURCE
rpool/ROOT/pve-1  acltype   off       default

That means ACL's are not on.

SOLUTION:

Enable ZFS POSIX ACLs:

Bash:

$ zfs set acltype=posixacl rpool/ROOT/pve-1

Check it again:

Bash:

$ zfs get acltype rpool/ROOT/pve-1

if it returns:

Bash:

NAME              PROPERTY  VALUE     SOURCE
rpool/ROOT/pve-1  acltype   posix     local

then Success!

Now try that LXC Backup again!

Credit goes to this guy: @CH.illig --> https://forum.proxmox.com/members/ch-illig.36347/ for his post (in German - thank you Google Translate):
https://forum.proxmox.com/threads/lxc-backup-fehler-wegen-acl.129309/

I hope this helps!

exitcomestothis · Feb 26, 2024

I just ran into this issue this afternoon after setting up my first install for PBS.

I have a few VM's running on my homelab and am curious what the ramifications would be of running this command?

Hopefully it doesn't cause any issues, but if it does, would the revert command be:

Code:

zfs set acltype=default rpool/ROOT/pve-1

Mateor · May 6, 2024

exitcomestothis said:
I just ran into this issue this afternoon after setting up my first install for PBS.

I have a few VM's running on my homelab and am curious what the ramifications would be of running this command?

Hopefully it doesn't cause any issues, but if it does, would the revert command be:

Code:

zfs set acltype=default rpool/ROOT/pve-1

I am also going to try this and wanted to know. I looked up the Man page (https://zfsonlinux.org/manpages/0.8.4/man8/zfs.8.html) here is what is listed.

acltype = off | noacl | posixacl Controls whether ACLs are enabled and if so what type of ACL to use.

off
default, when a file system has the acltype property set to off then ACLs are disabled.
noacl
an alias for off
posixacl
indicates POSIX ACLs should be used. POSIX ACLs are specific to Linux and are not functional on other platforms. POSIX ACLs are stored as an extended attribute and therefore will not overwrite any existing NFSv4 ACLs which may be set.

To obtain the best performance when setting posixacl users are strongly encouraged to set the xattr=sa property. This will result in the POSIX ACL being stored more efficiently on disk. But as a consequence, all new extended attributes will only be accessible from OpenZFS implementations which support the xattr=sa property. See the xattr property for more details.

so, I believe the correct revert command would be.

Code:

zfs set acltype=off rpool/ROOT/pve-1

or

Code:

zfs set acltype=noacl rpool/ROOT/pve-1

Search

Search

[SOLVED] LXC Backup faild ( exit code 23 )

demon_mono

Member

fabian

Proxmox Staff Member

demon_mono

Member

demon_mono

Member

AnK

Member

fabian

Proxmox Staff Member

AnK

Member

amorphis

New Member

fabian

Proxmox Staff Member

amorphis

New Member

amorphis

New Member

MaFL

Renowned Member

fabian

Proxmox Staff Member

mangelot

Member

fabian

Proxmox Staff Member

Zubin Singh Parihar

Well-Known Member

exitcomestothis

Active Member

Mateor

Member