Permission issues when administering unpriviliged containers

[dbn]

On a completely fresh install of Proxmox VE 5.1, I'm unable to create unprivileged containers from templates and creating backups. I get permission errors when I try.

The issue with creating the containers seems to be that templates gets saved with root ownership and strict permissions that doesn't allow the unprivileged user related to the unprivileged container to read it. This is explained in a few more details on my post on reddit which I cannot neither link to or post a direct screendump of as this forum won't allow that. Remove the spaces: i.imgur. com/ IFwcxkf.png

Another issue is that I'm unable to create backups of unprivileged containers. I guess this is also some variation of the issue described above. This is the log outputted:

Spoiler: backup log

INFO: starting new backup job: vzdump 100 --storage nas01-backup --remove 0 --node pxmx01 --mode snapshot --compress 0
INFO: Starting Backup of VM 100 (lxc)
INFO: status = running
INFO: CT Name: unifi
INFO: mode failure - some volumes do not support snapshots
INFO: trying 'suspend' mode instead
INFO: backup mode: suspend
INFO: ionice priority: 7
INFO: CT Name: unifi
INFO: temporary directory is on NFS, disabling xattr and acl support, consider configuring a local tmpdir via /etc/vzdump.conf
INFO: starting first sync /proc/26112/root// to /mnt/pve/nas01-backup/dump/vzdump-lxc-100-2018_03_13-18_00_09.tmp
INFO: Number of files: 27,336 (reg: 20,913, dir: 2,327, link: 4,066, special: 30)
INFO: Number of created files: 27,335 (reg: 20,913, dir: 2,326, link: 4,066, special: 30)
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 20,904
INFO: Total file size: 1,700,831,267 bytes
INFO: Total transferred file size: 1,693,646,152 bytes
INFO: Literal data: 1,693,646,378 bytes
INFO: Matched data: 0 bytes
INFO: File list size: 589,789
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 1,695,622,527
INFO: Total bytes received: 423,831
INFO: sent 1,695,622,527 bytes received 423,831 bytes 21,068,898.86 bytes/sec
INFO: total size is 1,700,831,267 speedup is 1.00
INFO: first sync finished (80 seconds)
INFO: suspend vm
INFO: starting final sync /proc/26112/root// to /mnt/pve/nas01-backup/dump/vzdump-lxc-100-2018_03_13-18_00_09.tmp
INFO: Number of files: 27,336 (reg: 20,913, dir: 2,327, link: 4,066, special: 30)
INFO: Number of created files: 0
INFO: Number of deleted files: 0
INFO: Number of regular files transferred: 3
INFO: Total file size: 1,700,831,492 bytes
INFO: Total transferred file size: 1,070,804 bytes
INFO: Literal data: 5,789 bytes
INFO: Matched data: 1,065,015 bytes
INFO: File list size: 0
INFO: File list generation time: 0.001 seconds
INFO: File list transfer time: 0.000 seconds
INFO: Total bytes sent: 680,375
INFO: Total bytes received: 8,892
INFO: sent 680,375 bytes received 8,892 bytes 196,933.43 bytes/sec
INFO: total size is 1,700,831,492 speedup is 2,467.59
INFO: final sync finished (3 seconds)
INFO: resume vm
INFO: vm is online again after 3 seconds
INFO: creating archive '/mnt/pve/nas01-backup/dump/vzdump-lxc-100-2018_03_13-18_00_09.tar'
INFO: tar: ./etc/vzdump/pct.conf: Cannot stat: Permission denied
INFO: tar: ./etc/vzdump: Cannot open: Permission denied
INFO: tar: ./home/admdanni/.bash_history: Cannot open: Permission denied
INFO: tar: ./var/cache/apt/archives/partial: Cannot open: Permission denied
INFO: tar: ./var/lib/apt/lists/partial: Cannot open: Permission denied
INFO: tar: ./var/lib/postfix/master.lock: Cannot open: Permission denied
INFO: tar: ./var/lib/unifi: Cannot open: Permission denied
INFO: tar: ./var/log/unifi: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/active: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/bounce: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/corrupt: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/defer: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/deferred: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/flush: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/hold: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/incoming: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/maildrop: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/private: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/public: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/saved: Cannot open: Permission denied
INFO: tar: ./var/spool/postfix/trace: Cannot open: Permission denied
INFO: Total bytes written: 1299312640 (1.3GiB, 28MiB/s)
INFO: tar: Exiting with failure status due to previous errors
ERROR: Backup of VM 100 failed - command 'set -o pipefail && lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar cpf - --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' --one-file-system '--warning=no-file-ignored' '--directory=/mnt/pve/nas01-backup/dump/vzdump-lxc-100-2018_03_13-18_00_09.tmp' ./etc/vzdump/pct.conf '--directory=/mnt/pve/nas01-backup/dump/vzdump-lxc-100-2018_03_13-18_00_09.tmp' --no-anchored '--exclude=lost+found' --anchored '--exclude=./tmp/?*' '--exclude=./var/tmp/?*' '--exclude=./var/run/?*.pid' . >/mnt/pve/nas01-backup/dump/vzdump-lxc-100-2018_03_13-18_00_09.dat' failed: exit code 2
INFO: Backup job finished with errors
TASK ERROR: job errors

I'm unsure how to go about this. I can find plenty of people on google having these issues, but no solutions.

Suggestions would be very much appreciated

 

[dietmar]

what version do you use exactly?

# pveversion -v

 

[wbumiller]

With unprivileged containers, the `tar` command runs as the unprivileged root user, these error suggest that those files are not accessible for that user. Can you please provide the output of the following executed from within the running container?

# ls -l /var/spool/postfix

Also, did you ever make any manual modifications to the container's filesystem (by mounting it manually or using `pct mount` on the host?)

 

[dbn]

what version do you use exactly?

# pveversion -v

Spoiler: pveversion -v

root@pxmx01:~# pveversion -v
proxmox-ve: 5.1-41 (running kernel: 4.13.13-6-pve)
pve-manager: 5.1-46 (running version: 5.1-46/ae8241d4)
pve-kernel-4.13.13-6-pve: 4.13.13-41
pve-kernel-4.13.13-2-pve: 4.13.13-33
corosync: 2.4.2-pve3
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-common-perl: 5.0-28
libpve-guest-common-perl: 2.0-14
libpve-http-server-perl: 2.0-8
libpve-storage-perl: 5.0-17
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 2.1.1-3
lxcfs: 2.0.8-2
novnc-pve: 0.6-4
proxmox-widget-toolkit: 1.0-11
pve-cluster: 5.0-20
pve-container: 2.0-19
pve-docs: 5.1-16
pve-firewall: 3.0-5
pve-firmware: 2.0-3
pve-ha-manager: 2.0-5
pve-i18n: 1.0-4
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.9.1-9
pve-xtermjs: 1.0-2
qemu-server: 5.0-22
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.6-pve1~bpo9

With unprivileged containers, the `tar` command runs as the unprivileged root user, these error suggest that those files are not accessible for that user. Can you please provide the output of the following executed from within the running container?

# ls -l /var/spool/postfix

Also, did you ever make any manual modifications to the container's filesystem (by mounting it manually or using `pct mount` on the host?)

No manual changes to filesystem.

Spoiler: ls -l /var/spool/postfix

root@unifi:/home/admdanni# ls -l /var/spool/postfix
total 72
drwx------ 2 postfix root 4096 Mar 13 05:27 active
drwx------ 2 postfix root 4096 Jun 19 2017 bounce
drwx------ 2 postfix root 4096 Jun 19 2017 corrupt
drwx------ 2 postfix root 4096 Jun 19 2017 defer
drwx------ 2 postfix root 4096 Jun 19 2017 deferred
drwxr-xr-x 2 root root 4096 Mar 14 15:05 dev
drwxr-xr-x 3 root root 4096 Mar 14 15:05 etc
drwx------ 2 postfix root 4096 Jun 19 2017 flush
drwx------ 2 postfix root 4096 Mar 13 05:01 hold
drwx------ 2 postfix root 4096 Mar 13 05:27 incoming
drwxr-xr-x 3 root root 4096 Mar 13 05:01 lib
drwx-wx--T 2 postfix postdrop 4096 Mar 13 05:27 maildrop
drwxr-xr-x 2 root root 4096 Mar 13 05:25 pid
drwx------ 2 postfix root 4096 Mar 14 15:05 private
drwx--s--- 2 postfix postdrop 4096 Mar 14 15:05 public
drwx------ 2 postfix root 4096 Jun 19 2017 saved
drwx------ 2 postfix root 4096 Mar 13 05:01 trace
drwxr-xr-x 3 root root 4096 Jun 19 2017 usr

 

[wbumiller]

The output you posted initially says it's using a temp-dir on an NFS storage. If your local disk space allows it, can you please try setting a `tmpdir` in /etc/vzdump.conf` to a directory on local storage? (Putting the temporary data on NFS has some drawbacks in addition to the extra bandwidth usage of copying back and forth multiple times over the net (ranging from lack of xattr & acl support to NFS specific permission quirks).

 

[dbn]

The output you posted initially says it's using a temp-dir on an NFS storage. If your local disk space allows it, can you please try setting a `tmpdir` in /etc/vzdump.conf` to a directory on local storage? (Putting the temporary data on NFS has some drawbacks in addition to the extra bandwidth usage of copying back and forth multiple times over the net (ranging from lack of xattr & acl support to NFS specific permission quirks).

This fixed the backup issue, but the template issue is still there. When I try to create a unprivileged container from a template downloaded through the proxmox gui it fails with this output:

Spoiler: output

Virtual Environment 5.1-46
You are logged in as 'root@pam'
()
Task viewer: CT 102 - Create
OutputStatus
Stop
Formatting '/mnt/pve/nas01-vmdata/images/102/vm-102-disk-1.raw', fmt=raw size=8589934592
mke2fs 1.43.4 (31-Jan-2017)
Creating filesystem with 2097152 4k blocks and 524288 inodes
Filesystem UUID: c9b28f6e-9773-4edb-a051-ae6a46851446
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: 0/64 done
Writing inode tables: 0/64 done
Creating journal (16384 blocks): done
Multiple mount protection is enabled with update interval 5 seconds.
Writing superblocks and filesystem accounting information: 0/64 done

extracting archive '/mnt/pve/nas01-templates/template/cache/debian-8.0-standard_8.7-1_amd64.tar.gz'
tar: /proc/self/fd/14: Cannot open: Permission denied
tar: Error is not recoverable: exiting now
TASK ERROR: command 'lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- tar xpf /proc/self/fd/14 --totals --one-file-system -p --sparse --numeric-owner --acls --xattrs '--xattrs-include=user.*' '--xattrs-include=security.capability' '--warning=no-file-ignored' '--warning=no-xattr-write' -C /var/lib/lxc/102/rootfs --skip-old-files --anchored --exclude './dev/*'' failed: exit code 2

 

[dbn]

Can I somehow change what permissions Proxmox assigns to freshly downloaded templates?

 

[dbn]

@wbumiller @dietmar
Would you be able to advise as to how to solve this? I’m hoping to roll out proxmox in my two businesses to replace xenserver soon, but since we rely on NFS storage to work for templates, vm data and backups, i need to make sure it works before buying a license/subscription.

 

[wbumiller]

There's a fix for opening template archives as root when creating unprivileged containers on the way currently. Should hit the test repositories soon.

 

[dbn]

@wbumiller Is it a fix I can implement myself now without switching to test repo's by any chance? Or would it help to make a temporarily workaround, like switching the templates storage from NFS to SMB or something else?

 

[wbumiller]

Mh, you might be able to get around this with an smb/cifs mount setting dir_mode and file_mode to allow reading of all files to all users and use that for templates if that's less of a hassle for you than trying pve-container>=2.0-21 from pvetest. For smb/cifs you'd have to mount it manually and use a directory storage since the cifs storage plugin has only recently been added and will have its initial release with libpve-storage-perl >= 5.0-18 which is not yet on the public test repository.

 

[dbn]

@wbumiller
I got it working with a CIFS mount. I bumped into a few quirks on the way, not sure if they're intentional or not tho. The CIFS mounted fine without any need to add dir_mode and file_mode options to fstab. However, it wouldn't auto-remount on reboot because the network interfaces were brought up AFTER parsing fstab. The fstab _netdev option is supposed to take care of that by delaying mounts of network devices till after network have been brought up, but that didn't work because systemd-networkd-wait-online.service wasn't enabled by default for some reason. I enabled that and it mounted on reboot as expected. As mentioned I'm not sure if this is intentional or not, but maybe it's something to consider for a future release

Thank you for your help.