CT restore problem with ZFS

Gilles

Active Member
Dec 4, 2012
19
1
28
Hi,

On PM4.4-18 I had to do a CT restore from backups scheduled with the GUI :

- stop the CT
- choose the good archive file
- launch restore

Then the CT totally disappear from the GUI and do not work anymore with the following message :

extracting archive '/mnt/pve/backup/dump/vzdump-lxc-100-2017_08_24-02_02_01.tar.gz'
tar: ./.zfs/snapshot/avant_1: Cannot mkdir: Permission denied
tar: ./.zfs/snapshot/rep_2017-03-20_12\:30\:04: Cannot mkdir: Permission denied
Total bytes read: 4031334400 (3.8GiB, 93MiB/s)
tar: Exiting with failure status due to previous errors
TASK ERROR: command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf /mnt/pve/backup/dump/vzdump-lxc-100-2017_08_24-02_02_01.tar.gz --totals --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-xattr-write' -C /var/lib/lxc/100/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

It seems the problem is related to the fact that the CT had snapshots (manual). I did not know that a CT was saved with its snapshots. By the way I did not find how to restore my CT !

In case of problem PM should allow to go back to the working CT, but I am even not able to restore this last version of the CT.... is Proxmox CT backup/restore reliable on ZFS ?


Thanks for your help
 
Last edited:

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
503
103
Hi,

I can't reproduce it here.

please send the output of

pveversion -v
pct config <VMID>
 

Gilles

Active Member
Dec 4, 2012
19
1
28
Hi Wolfgang,

pveversion - v :
proxmox-ve: 4.4-94 (running kernel: 4.4.44-1-pve)
pve-manager: 4.4-18 (running version: 4.4-18/ef2610e8)
pve-kernel-4.4.35-1-pve: 4.4.35-77
pve-kernel-4.4.59-1-pve: 4.4.59-87
pve-kernel-4.4.44-1-pve: 4.4.44-84
pve-kernel-4.4.67-1-pve: 4.4.67-92
pve-kernel-4.4.76-1-pve: 4.4.76-94
pve-kernel-4.4.49-1-pve: 4.4.49-86
pve-kernel-4.4.62-1-pve: 4.4.62-88
lvm2: 2.02.116-pve3
corosync-pve: 2.4.2-2~pve4+1
libqb0: 1.0.1-1
pve-cluster: 4.0-52
qemu-server: 4.0-112
pve-firmware: 1.1-11
libpve-common-perl: 4.0-96
libpve-access-control: 4.0-23
libpve-storage-perl: 4.0-76
pve-libspice-server1: 0.12.8-2
vncterm: 1.3-2
pve-docs: 4.4-4
pve-qemu-kvm: 2.7.1-4
pve-container: 1.0-101
pve-firewall: 2.0-33
pve-ha-manager: 1.0-41
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u3
lxc-pve: 2.0.7-4
lxcfs: 2.0.6-pve1
criu: 1.6.0-1
novnc-pve: 0.5-9
smartmontools: 6.5+svn4324-1~pve80

pct config 100 : Configuration file 'nodes/XXXXX/lxc/100.conf' does not exist

The CT has simply vanished. I have to say I use the ayufan.eu differential backup script.

Gilles
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
503
103
I would contact ayufan.
He use another backup format then we does.
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
503
103
You should contact him.
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
503
103
Can you please check if you have enable snapdir on this dataset?

zfs get snapdir <zfspool>
 

Gilles

Active Member
Dec 4, 2012
19
1
28
zfs get snapdir rpool :
NAME PROPERTY VALUE SOURCE
rpool snapdir visible local

By the way, for those who use this nice piece of patch from Ayufan (differential backup)
I have solved the problem by adding another --exclude './.zfs/.*' in the tar xpf (Create.pm).

I did not find any solution to be able to create some dir or files in /.zfs
 

wolfgang

Proxmox Retired Staff
Retired Staff
Oct 1, 2014
6,496
503
103
I did not find any solution to be able to create some dir or files in /.zfs
This is not possible to write in this, they are the snapshots and snapshot are not writable in zfs.
 

remyd1

New Member
Apr 24, 2017
3
0
1
41
Hi,

I encountered a very similar problem; I was able to enter in the tarball archive, but I get "Permission Denied" for each file in my archive. My storage is also Zfs as a backend.
After googling a lot, I have been able to solve my issue.

The "pct restore" script is calling "lxc-usernsexec" from LXC ( /usr/share/perl5/PVE/LXC.pm ) with subuid and subgid 100000. However a basic "ls -aln" inside a new namespace from "lxc-usernsexec -- /bin/bash" showed me nobody:nogroup (in container using lxc-usernsexec) for every root permissions (when connecting to the running container or using lxc-attach). Indeed uid and gid are set to 65534... You can check all these things using lsns, ps, and by editing /usr/share/perl5/PVE/LXC.pm.

If you try anything with your new user, you will see that you can't do anything...

However, if we stay with the standards /etc/{subuid, subgid}, it is not possible to get such an uid/gid inside the container. Indeed, normally, /etc/subuid and /etc/subgid match root as :
root:100000:65536
That means root will get a sub{gid,uid} between 100000 and 165536 for each new namespace. So the new root inside the container won't be able to restore our archive, because he will never get the nobody/nogroup uid/gid (65534).

setfacl can easily solve this issue :

# let's say your new container id is 110
CTID=110
zfs create zfspool/$CTID -m /where/your/want/to/restore/$CTID
cd /where/your/want/to/restore
setfacl -m user:100000:rwX .
setfacl -R -m user:100000:rwX vzdump-lxc-...tar.lzo
setfacl -R -m user:100000:rwX $CTID
# some basic checking
getfacl .
ls -all .
ls -al $CTID && getfacl $CTID
pct restore $CTID vzdump-lxc-....tar.lzo -rootfs volume=/where/your/want/to/restore/$CTID -storage zfspool -ignore-unpack-errors 1 -unprivileged​
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!