pct restore failed: exit code 2 on Proxmox Cluster

[TECOB]

Hi,

Don't know why but we can't restore a vzdump of an LXC container into a Proxmox cluster composed of 6 nodes with ZFS.

This is the situation:
We have a Proxmox Cluster with 6 nodes, each one running with ZFS. Each night dumps are made of al containers and VM and set fia FTP and SCP to different locations.
We wanted to restore an LXC Container dump to the cluster with another ID. The real LXC container (ID 234) is still runnign in the cluster and we are restoring the dump of 2 days ago with the ID 99X.

Each time we try to restor we get an error and it's impossible to restore it.

This is what we get:

extracting archive '/var/lib/vz/dump/vzdump-lxc-234-2019_10_16-04_28_05.tar.lzo'
tar: ./var/spool/postfix/dev/urandom: Cannot mknod: Operation not permitted
tar: ./var/spool/postfix/dev/random: Cannot mknod: Operation not permitted
Total bytes read: 64203161600 (60GiB, 150MiB/s)
tar: Exiting with failure status due to previous errors
TASK ERROR: unable to restore CT 999 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --lzop --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/999/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

The same if we try to restore to another ID and even with the dump of 1 day ago.

Instead, we moved the tar.lzo file to another Proxmox server ( no ZFS) outside the cluster and we made a restore. Everything runned ok and the vzdump with ID 999 is restored and we could recover files.

We checked logs of vzdump and both days vzdump runned ok and dump was made ok.

The question is, why can't we restore an old vzdump into the cluster using another ID ?

I paste here versions of pve-*

pve-cluster 5.0-37
pve-container 2.0-39
pve-docs 5.4-2
pve-edk2-firmware 1.20190312-1
pve-firewall 3.0-21
pve-firmware 2.0-6
pve-ha-manager 2.0-9
pve-i18n 1.1-4
pve-kernel-4.15 5.4-2
pve-kernel-4.15.18-14-pve 4.15.18-39
pve-libspice-server1 0.14.1-2
pve-manager 5.4-6
pve-qemu-kvm 3.0.1-2

Also, on each node cluster replication is made every 5 minutes of each node in another node, in a circular way.
Logs about dumps are clear and don't throw any error, and furthermore, dump could be restored in a Proxmox server out of the Cluster.

could someone tell us what are we doing wrong?

Regards,

 

[Chris]

Hi,
please try to restore as privileged container (checkbox in the dialog). Device nodes are not allowed for unprivileged containers.

 

[TECOB]

Hi Chris,

OMG you're right. Stupid me. Sorry for that mistake.

Best regarsd

 

[Chris]

Hi Chris,

OMG you're right. Stupid me. Sorry for that mistake.

Best regarsd

No worries! Glad it works

 

[the.crusher]

Hi Chris,

Can you help me with a related question?
I have a single node setup with an unprivileged LXC container. I have backups since the beginning, and I restored it a few times without problem (as unprivileged container). But yesterday I had to restore again, but I had some errors similar to OP's:

tar: ./var/lib/docker/overlay2/558ead5455f0a70049ac9f0dd53545d87328eb3f2664d3712051e3b41e55dd09/diff/usr/src/requirements.txt: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/de6ca77d2468f5d0d19b124a9fa75bdd8c512681370185d5ac8062277a8412e5/diff/usr/src/requirements.txt: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/059f8bcf639b45e168a8d14906251304abf8648bdb0c9f7cb8c618714fef4793/diff/usr/src/fix-xattrs-glibc.patch: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/059f8bcf639b45e168a8d14906251304abf8648bdb0c9f7cb8c618714fef4793/diff/usr/src/musl-find_library.patch: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/e5327b701d88a4ab65222c216137a46d5575b9451df7486f4542732e86e40ddf/diff/usr/src/arm-alignment.patch: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/e5327b701d88a4ab65222c216137a46d5575b9451df7486f4542732e86e40ddf/diff/usr/src/musl-find_library.patch: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/e5327b701d88a4ab65222c216137a46d5575b9451df7486f4542732e86e40ddf/diff/usr/src/fix-xattrs-glibc.patch: Cannot mknod: Operation not permitted
tar: ./var/lib/docker/overlay2/0f37ac99d9e0e9ec8445b4f4c5054c193499cfe53e2eb60ddbd3a11f543a8807/diff/usr/src/requirements.txt: Cannot mknod: Operation not permitted

tar: Exiting with failure status due to previous errors
TASK ERROR: unable to restore CT 103 - command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf - --zstd --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/103/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

The solution was what you mentioned, make it privileged. But earlier I did not have to make it privileged. Is it possible to restore as unprivileged? is it because some updates in PVE? Or it is related to the change in compression I made recently (sounds unlikely, but who knows) from GZIP to ZSTD? Or maybe I made some changes in the container?

Thanks!