Error restoring LXC with nested,keyctl options

[Alex L]

I encountered an error restoring an unprivileged LXC container with the nesting=1,keyctl=1 options set (for use with Docker). Here's the relevant part of the log:

extracting archive '/tank0/enc/proxmox/dump/vzdump-lxc-105-2020_07_09-20_34_11.tar.zst'

tar: ./var/lib/docker/overlay2/83740d2e44ae84691ea534a7ea046219bbd8d06e15c75c5a1494024715989e1b/diff/usr/lib/python3.7/site-packages/Radicale-2.1.12.dist-info: Cannot mknod: Operation not permitted

Total bytes read: 4846110720 (4.6GiB, 240MiB/s)

tar: Exiting with failure status due to previous errors

  Logical volume "vm-105-disk-0" successfully removed

TASK ERROR: unable to restore CT 105 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/105/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

I've tried restoring as a privileged container (it works), but adding unprivileged: 1 into the .conf file means I'm no longer able to log in. Any ideas, short of dumping the container's data and creating a new one? Thanks.

 

[Alex L]

Correction: restoring to a privileged container completes without error, but the container is unusable even when left as privileged. I can log in, but I get a bunch of errors when trying to start a Docker container from AppArmor.

I found a convoluted workaround by extracting the container backup and deleting Docker's overlay2 directory, which works at the cost of erasing Docker's image repository and active containers. If you can pull images from a repository or rebuild them from source, it's an option. Hopefully the Proxmox folks can help with a better alternative or fix the bug.

mkdir tmp
cd tmp
tar -I zstd -xf /var/lib/vz/dump/container-backup.tar.zst
rm -rf var/lib/docker/overlay2
tar cf restore.tar .
/usr/sbin/pct restore 105 restore.tar --unprivileged 1 --features keyctl=1,nesting=1

 

[Stoiko Ivanov]

Not sure I get the complete picture:
Was the container originally unprivileged ? (sounds doubtful, since unprivileged containers cannot create device nodes)
I would expect it was privileged and thus needs to be restored as privileged container (simply changing the config by adding 'unprivileged: 1' will not work)

 

[Alex L]

Yes, the container was originally unprivileged. Following my restore instructions above, I was able to get it back up and running, but if I do another backup and restore from the restored container, the same issue occurs. I'm running Proxmox 6.2-9.

However, if I create a new container with the same settings, rsync all the user data over, and do a backup/restore, it works properly. I tested various hunches (was it related to Docker container builds, since the issue appears to be the overlay2 file system?), but I was unable to reproduce the bug. My original container was created in an earlier version of Proxmox, using older Docker etc., so perhaps there was an issue somewhere there. But whatever it was, it seems to not be an issue anymore. My apologies for being unable to help diagnose this further.

 

[Stoiko Ivanov]

Hmm - curious - but well - if you cannot reproduce it - I guess we'll leave it at that.

Thanks for reporting back!

 

[tripflex]

For anybody coming here later and seeing this, if the container is unprivileged just goto Options from the container menu (main UI), click on Features then select keyctl and nesting, save, voila!