Cannot restore LXC container from PBS to PVE

[snek]

I have 3 PBS servers, VM's and LXC containers backup to PBS1 then they are synchronized to PBS2. Lastly they backup to offsite PBS3.
However I cannot restore LXC container from any of the PBS.

The error i get is:

Error: error extracting archive - error at entry "": failed to apply directory metadata: failed to apply acls: Error while restoring ACL - ACL invalid
  Logical volume "vm-108-disk-0" successfully removed
TASK ERROR: unable to restore CT 108 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client restore '--crypt-mode=none' ct/108/2021-03-29T22:05:04Z root.pxar /var/lib/lxc/108/rootfs --allow-existing-dirs --repository ***@***@123.123.123.123:BACKUP' failed: exit code 255

Everything was fine few days ago, so its probably because of some new update.

Spoiler: output of pveversion -v

proxmox-ve: 6.3-1 (running kernel: 5.4.106-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.103-1-pve: 5.4.103-1
pve-kernel-5.4.98-1-pve: 5.4.98-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.11-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.2.0-4
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-8
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

 

[nakaori]

Hi snek,

I posted the same earlier this morning:

https://forum.proxmox.com/threads/failed-to-apply-acls.86885/

Maybe you can replicate as fabian said in this thread? I wasn't able to reproduce it with an empty container.

 

[snek]

Couldn't reproduce it with empty container either. I had to restore and older backup (from 24.3.2021) which worked fine.

 

[snek]

I just tried to backup again the old restored backup. Still not able to restore it. I updated the PVE and PBS. It seems that i am unable to restore any LXC.

 

[nakaori]

i am doing the same atm. but instead of backing up to pbs i am backing up locally. will let you know.