Unable to restore LXC backup (ACL invalid)

Finski

New Member
Dec 11, 2022
3
0
1
To confirm that my backup arrangement is working, I've been trying to do an inplace recovery of an LXC backup but, no matter the settings I try (either from GUI or CLI) the backup recovery keeps failing with:

Code:
Error: error extracting archive - error at entry "": failed to apply directory metadata: failed to apply acls: Error while restoring ACL - ACL invalid

The target LXC is unprivileged, has one mp and GPU passthrough enabled. I've tried to do the recovery in the CLI with:

Code:
pct restore 102 local_pbs:backup/ct/102/2022-12-09T23:00:53Z --rootfs ncp_data:800 --mp0 ncp_archive:1500,mp=/data_archive --ignore-unpack-errors=true

I've tried restoring the backup with unprivileged set to both =1 and =0.

I've also tried the recovery of another LXC on the same system and that worked flawlessly. The biggest difference between these containers is that the problematic container has GPU passthrough enabled. Can that be behind these issues?

Here's the full output from the failed recovery, when trying to do the recovery from the GUI to a privileged container (don't mind about the thin volume size warning, that's irrelevant in this case as the recovered volumes are only about ~1TB in size):
Code:
recovering backed-up configuration from 'local_pbs:backup/ct/102/2022-12-09T23:00:53Z'
  Logical volume "vm-102-disk-0" created.
Creating filesystem with 209715200 4k blocks and 52428800 inodes
Filesystem UUID: f3a50afc-edbd-4b14-b4ee-89b99c3d57d5
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "vm-102-disk-1" created.
  WARNING: Sum of all thin volume sizes (<2.25 TiB) exceeds the size of thin pool ncp_archive/ncp_archive and the size of whole volume group (<1.82 TiB).
Creating filesystem with 393216000 4k blocks and 98304000 inodes
Filesystem UUID: e07a8265-3d74-434d-9e7e-b3f513e28075
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000, 214990848
restoring 'local_pbs:backup/ct/102/2022-12-09T23:00:53Z' now..
Error: error extracting archive - error at entry "": failed to apply directory metadata: failed to apply acls: Error while restoring ACL - ACL invalid
  Logical volume "vm-102-disk-0" successfully removed
  Logical volume "vm-102-disk-1" successfully removed
TASK ERROR: unable to restore CT 102 - command '/usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/102/2022-12-09T23:00:53Z root.pxar /var/lib/lxc/102/rootfs --allow-existing-dirs --repository user_name@pbs@***.***.***.***:local_hdd' failed: exit code 255

PVE version: 7.3-3 (non-subscription)
PBS vesrion: 2.3-1 (non-subscription)

I'm a complete Proxmox newbie and at a loss here about where to start even looking for the issue. Could somebody, please, point me in the right direction so as to find out what I've screwed up? I know I can mount the backup and manually restore the files to a new container but I'd much rather have the automatic recovery working as I've, unfortunately, picked up the habit of frequently nuking my "production" containers.

Thanks in advance for any help!
 
Last edited:
Could it be that you are using centos7 container with custom systemd version?
Thanks for the response! Unfortunately, no, it's a debian container running Nextcloud. Here's the conf:

Code:
arch: amd64
cores: 4
features: nesting=1,keyctl=1
hostname: nextcloud
memory: 10240
mp0: ncp_archive:vm-102-disk-0,mp=/data_archive,backup=1,size=1500G
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.201.1,hwaddr=14:CF:44:DA:C9:A8,ip=192.168.201.180/24,type=veth
net1: name=net1,bridge=vmbr1,hwaddr=8B:44:13:1F:FA:82,ip=192.168.202.202/24,type=veth
onboot: 1
ostype: debian
rootfs: ncp_data:vm-102-disk-0,size=800G
swap: 2048
unprivileged: 1
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
 
I tried restoring the files manually using rsync to an empty container. Several permission errors (e.g. log files) prevented redis-server from starting, but that was quick to fix. Other than that everything seems to work in the backup.

Could this have been caused due to the backup being based on a snapshot?
 
Last edited:
Hi
I have the same problem...

Ubuntu container with nextcloud.
It is always the same file which cause the error after ~1,5h -> "00039.MTS"
I manually (via FileRestore) was able to restore the "00039.MTS" from the backup without any problems and file was not corrupted.
I also tried other/previous backups (which are done via snapshot backup) - same result
Then I stopped the container and created a backup with container stopped, same result


Any additonal logs, which can give further information?
Any hints? - thanks

UPDATE: Just realized that the proxmox-backup-client version is not the same on both machine - will update and test with same versions

---- Futher Infos ---

UPDATE: Add versions
root@pve0:~# proxmox-backup-client version
client version: 2.2.7

root@pve1:~# proxmox-backup-client version
client version: 2.3.1

promox version on both is 7.3-3

Restore is done on another proxmox instance, than the backuped container is running on.

Tested backups are verified OK

Code:
#Ubuntu 22.04
arch: amd64
console: 1
cores: 14
hostname: mynextcloud
memory: 16384
mp0: local-hdd-1:vm-109-disk-0,mp=/data,backup=1,size=1200G
net0: name=eth0,bridge=vmbr0,firewall=1,gw=....,hwaddr=::::,ip=...../24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-109-disk-0,size=16G
swap: 8192
unprivileged: 1



Code:
recovering backed-up configuration from 'pve0-pbs-hdd:backup/ct/109/2023-01-06T23:05:42Z'
  Logical volume "vm-105-disk-0" created.
Creating filesystem with 4194304 4k blocks and 1048576 inodes
Filesystem UUID: 3b6c691c-3435-4684-a84a-0fea25b34458
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
  WARNING: You have not turned on protection against thin pools running out of space.
 WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "vm-105-disk-1" created.
 WARNING: Sum of all thin volume sizes (<1.19 TiB) exceeds the size of thin pool local-hdd-1/local-hdd-1 and the size of whole volume group (931.51 GiB).
Creating filesystem with 314572800 4k blocks and 78643200 inodes
Filesystem UUID: 13c16447-ac04-41a9-b768-bed733a3adac
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000, 214990848
restoring 'pve0-pbs-hdd:backup/ct/109/2023-01-06T23:05:42Z' now..
Error: error extracting archive - error at entry "00039.MTS": failed to apply acls: Error while restoring ACL - ACL invalid
  Logical volume "vm-105-disk-0" successfully removed
  Logical volume "vm-105-disk-1" successfully removed
TASK ERROR: unable to restore CT 105 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/109/2023-01-06T23:05:42Z root.pxar /var/lib/lxc/105/rootfs --allow-existing-dirs --repository @pbs@......:4TBext4' failed: exit code 255
 
Last edited:
ok also after update, creating a fresh backup - same problem

Error: error extracting archive - error at entry "00039.MTS": failed to apply acls: Error while restoring ACL - ACL invalid
 
Ok I created now a backup not using PBS, instead I did the backup via proxmox directly on an external NFS share and with this backup I was able to restore on the other proxmox instance without any errors - so it looks to be a problem related with the pbs backup method.
 
Last edited:
I was using rsync and I receive some "rsync: [generator] set_acl: sys_acl_set_file Invalid argument (22)" errors
Triggered by this I observed a difference when using getfacl for the failing file vs. others
I can remove the difference by using "setfacl -bn" But to be frank I'm lost what this actually means or what I'm doing here?
thx if someone can bring light into my tunnel :)

# file: gemeinsam/files/photo/00039.MTS
# owner: www-data
# group: www-data
user::rwx
user:4294967295:rwx
group::r-x
mask::rwx
other::r-x

vs. other files
# owner: www-data
# group: www-data
user::rwx
group::r-x
other::r-x
 
Last edited:
Update: After I removed all those strange ACL settings. A new created PBS Backups can be restored again without showing the error

PS: Feels like a monolog here ;-)
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!