CT restore problem with ZFS

[Gilles]

Hi,

On PM4.4-18 I had to do a CT restore from backups scheduled with the GUI :

- stop the CT
- choose the good archive file
- launch restore

Then the CT totally disappear from the GUI and do not work anymore with the following message :

extracting archive '/mnt/pve/backup/dump/vzdump-lxc-100-2017_08_24-02_02_01.tar.gz'
tar: ./.zfs/snapshot/avant_1: Cannot mkdir: Permission denied
tar: ./.zfs/snapshot/rep_2017-03-20_12\:30\:04: Cannot mkdir: Permission denied
Total bytes read: 4031334400 (3.8GiB, 93MiB/s)
tar: Exiting with failure status due to previous errors
TASK ERROR: command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf /mnt/pve/backup/dump/vzdump-lxc-100-2017_08_24-02_02_01.tar.gz --totals --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-xattr-write' -C /var/lib/lxc/100/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

It seems the problem is related to the fact that the CT had snapshots (manual). I did not know that a CT was saved with its snapshots. By the way I did not find how to restore my CT !

In case of problem PM should allow to go back to the working CT, but I am even not able to restore this last version of the CT.... is Proxmox CT backup/restore reliable on ZFS ?


Thanks for your help

 

[wolfgang]

Hi,

I can't reproduce it here.

please send the output of

pveversion -v
pct config <VMID>

 

[Gilles]

Hi Wolfgang,

pveversion - v :

proxmox-ve: 4.4-94 (running kernel: 4.4.44-1-pve)
pve-manager: 4.4-18 (running version: 4.4-18/ef2610e8)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.76-1-pve: 4.4.76-94
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-112
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80​

pct config 100 : Configuration file 'nodes/XXXXX/lxc/100.conf' does not exist

The CT has simply vanished. I have to say I use the ayufan.eu differential backup script.

Gilles

 

[wolfgang]

I would contact ayufan.
He use another backup format then we does.

 

[Gilles]

You will contact him or shall I ?

 

[wolfgang]

You should contact him.

 

[wolfgang]

Can you please check if you have enable snapdir on this dataset?

zfs get snapdir <zfspool>

 

[Gilles]

zfs get snapdir rpool :

NAME PROPERTY VALUE SOURCE
rpool snapdir visible local​

By the way, for those who use this nice piece of patch from Ayufan (differential backup)
I have solved the problem by adding another --exclude './.zfs/.*' in the tar xpf (Create.pm).

I did not find any solution to be able to create some dir or files in /.zfs

 

[wolfgang]

I did not find any solution to be able to create some dir or files in /.zfs

This is not possible to write in this, they are the snapshots and snapshot are not writable in zfs.

 

[remyd1]

Hi,

I encountered a very similar problem; I was able to enter in the tarball archive, but I get "Permission Denied" for each file in my archive. My storage is also Zfs as a backend.
After googling a lot, I have been able to solve my issue.

The "pct restore" script is calling "lxc-usernsexec" from LXC ( /usr/share/perl5/PVE/LXC.pm ) with subuid and subgid 100000. However a basic "ls -aln" inside a new namespace from "lxc-usernsexec -- /bin/bash" showed me nobody:nogroup (in container using lxc-usernsexec) for every root permissions (when connecting to the running container or using lxc-attach). Indeed uid and gid are set to 65534... You can check all these things using lsns, ps, and by editing /usr/share/perl5/PVE/LXC.pm.

If you try anything with your new user, you will see that you can't do anything...

However, if we stay with the standards /etc/{subuid, subgid}, it is not possible to get such an uid/gid inside the container. Indeed, normally, /etc/subuid and /etc/subgid match root as :
root:100000:65536
That means root will get a sub{gid,uid} between 100000 and 165536 for each new namespace. So the new root inside the container won't be able to restore our archive, because he will never get the nobody/nogroup uid/gid (65534).

setfacl can easily solve this issue :

# let's say your new container id is 110
CTID=110
zfs create zfspool/$CTID -m /where/your/want/to/restore/$CTID
cd /where/your/want/to/restore
setfacl -m user:100000:rwX .
setfacl -R -m user:100000:rwX vzdump-lxc-...tar.lzo
setfacl -R -m user:100000:rwX $CTID
# some basic checking
getfacl .
ls -all .
ls -al $CTID && getfacl $CTID
pct restore $CTID vzdump-lxc-....tar.lzo -rootfs volume=/where/your/want/to/restore/$CTID -storage zfspool -ignore-unpack-errors 1 -unprivileged​

 

[gomu]

Thanks for the workaround, it worked for me, but I find it rather complicated.

setfacl -m user:100000:rwX /where/your/want/to/restore/$CTID

was actually enough for me, the trick is to do it during the restore.
See my thread unable to restore an unprivileged CT as unprivileged