Writing to Bind Mount Inside of an Unprivileged Container?

[Davidoff]

Having a bit of trouble attempting to enable users of a group within a container to write to a bind mount. I'm on Proxmox 5.1 and created a standard Ubuntu 17.10 container from the available template. I tried using the PVE page on unprivileged containers as a guide in attempting to enable write access and took the following steps:

• In /etc/pve/lxc/100.conf I added mp0: /vdata/media,mp=/media
• Changed the container UID mapping in the file /etc/pve/lxc/100.conf:

lxc.idmap = u 0 100000 2000
lxc.idmap = g 0 100000 2000
lxc.idmap = u 2000 2000 1
lxc.idmap = g 200 2000 1
lxc.idmap = u 2001 102001 63535
lxc.idmap = g 2001 102001 63535

• Added root:2000:1 to /etc/subuid and /etc/subgid on host to allow root to use the uid/gid in the container
• Created a new group called container-data and add root to the group:
  

  addgroup --gid 2000 container-data
  usermod -aG container-data root

• Change group ownership on the host directory to container-data
  

  chgrp -R container-data /vdata/media
  chmod g+w /vdata/media
  chmod g+s /vdata/media

  
  
• Inside the container, created a container-data group with the same GID and added users to it:
  

  addgroup --gid 2000 container-data
  usermod -aG container-data root
  usermod -aG container-data service
  usermod -aG container-data david

Once all the above is done, if I enter the container and try to write in /media as root in the container, I get "Permission denied". Same if I su into another user (e.g. service or david). The only way I seem to be able to write to the bind mount from within the container is when I set the owner of the directory on the host to be a valid user in the container. For example, if I change the owner of /vdata/media to 100000 (i.e. root on the container) and then enter into the container (as root) then I can write to the /media just fine. However, no other user in the container-data group can do so.

I also previously tried to get group write rights in the container in a simpler manner without any of the above remapping entries (more for testing) - by just changing the group and owner on the host to a UID and GID mapped from the container - i.e. tried changing ownership to, for example 100000:100117 on the host (which corresponded to 0 - root and 117 - testgroup in the container). Same results - in the container root can write just fine, but no other member of testgroup could.

I guess another way of stating this is the group write permissions seem to be ignored in the container. Not sure what I'm doing wrong. Any suggestions would be most appreciated.

 

[wbumiller]

Is /vdata/media on NFS? Works here on ext4/zfs/btrfs, but I can reproduce this problem with NFS... Some file systems have additional checks which aren't always properly user namespace compatible. Seems NFS is currently problematic there.

 

[Davidoff]

Thanks for the note Wolfgang. No, /vdata/media is not on NFS. It's just a directory on a hard drive directly attached to the single node, formatted with ext4. Hmmm. If it works for you I must be missing something in my setup, but for the life of me can't quite figure it out. If you have any suggestions on how I might diagnose I would very much appreciate them.

In the unlikely event someone else is experiencing this (and I suspect there might be - I came across some threads on the Great Interwebs that seemed to suggest that this was a problem - I'd post a link but apparently can't because I'm a new user) I came up with a somewhat kludgy workaround: I use setfacl on the host to grant container users the necessary rights - for example, to grant write rights to root in the container (UID 0 in the container; 100000 on host), I do the following on the host:

setfacl -Rm user:100000:rwx,default:user:100000:rwx

Not ideal but it seems to work for the time being.

 

[ObsidianBlk]

Thanks for the note Wolfgang. No, /vdata/media is not on NFS. It's just a directory on a hard drive directly attached to the single node, formatted with ext4. Hmmm. If it works for you I must be missing something in my setup, but for the life of me can't quite figure it out. If you have any suggestions on how I might diagnose I would very much appreciate them.

In the unlikely event someone else is experiencing this (and I suspect there might be - I came across some threads on the Great Interwebs that seemed to suggest that this was a problem - I'd post a link but apparently can't because I'm a new user) I came up with a somewhat kludgy workaround: I use setfacl on the host to grant container users the necessary rights - for example, to grant write rights to root in the container (UID 0 in the container; 100000 on host), I do the following on the host:

setfacl -Rm user:100000:rwx,default:user:100000:rwx

Not ideal but it seems to work for the time being.

This post has been here a little while, but I think I solved the issue you had.
On my system I did everything you did and was finding the same issue until, just for fun, on the host I entered the following to my /mnt/share folder:

chown -R 2000 /mnt/share

Once I did that, I was able to add files to the share folder from within the container with any user I assigned my "share-container" group to.
Hope that helps someone!

 

[Davidoff]

Interesting. Thanks very much for the info. That's very helpful information indeed. I might give that a try at some point but ended up getting heavily invested in the setfacl approach, which seems to work well enough for my needs, at least for the time being.

 

[sigo]

Having a bit of trouble attempting to enable users of a group within a container to write to a bind mount. I'm on Proxmox 5.1 and created a standard Ubuntu 17.10 container from the available template. I tried using the PVE page on unprivileged containers as a guide in attempting to enable write access and took the following steps:

• In /etc/pve/lxc/100.conf I added mp0: /vdata/media,mp=/media
• Changed the container UID mapping in the file /etc/pve/lxc/100.conf:

lxc.idmap = u 0 100000 2000
lxc.idmap = g 0 100000 2000
lxc.idmap = u 2000 2000 1
lxc.idmap = g 200 2000 1
lxc.idmap = u 2001 102001 63535
lxc.idmap = g 2001 102001 63535

lxc.idmap = g 200 2000 1
or
lxc.idmap = g 2000 2000 1 ?

 

[Davidoff]

lxc.idmap = g 200 2000 1
or
lxc.idmap = g 2000 2000 1 ?

Thanks sigo - yes that was a typo in the post.

 

[Davidoff]

Sorry to revive an old thread. I recently upgraded to 6.0 and have started to encounter problems with permissions once again. Just to recap, I am using mount points to allow access within containers to certain directors on a physical drive attached to a single node and am using setfacl at the host level to assign rights to read, write and/or execute to users within the container. So, for example, I use the following to allow root in a container to access the directory /vdata (which is mounted in the container as /vdata as well)

setfacl -Rm user:100000:rwx,default:user:100000:rwx /vdata

When I enter the container and run getfacl on /vdata, I can see that root has rwx rights to /vdata. But when I run touch test while in the container, I get:

touch: cannot touch 'test': Permission denied

Within the container, /vdata is shown as being owned by nobody:nogroup. At the host level, it's root:root. Needless to say, I can as root write to the directory at the host level.

This seems to be something consistent across all containers - i.e. no users in any of the containers can now write to any of the bind mounted containers. It was all working fine prior to the update to 6.

I'm at a bit of a loss in figuring out how to further diagnose and/or fix this issue. If anyone has any suggestions, they would be most appreciated.

 

[oguz]

hi,

This seems to be something consistent across all containers - i.e. no users in any of the containers can now write to any of the bind mounted containers. It was all working fine prior to the update to 6.

did you by chance restore all your containers on the gui after the upgrade?

we changed the default to 'unprivileged' container during ct create/restore. that may be the reason you see nobody:nogroup from inside the container, because they run unprivileged and you didn't map the uid/gid.

check your container configs in /etc/pve/lxc/CT.conf

if you see `unprivileged: 1` and your containers were not unprivileged previously, that might as well be the cause. you can convert your cts to privileged if you make a backup and restore after unchecking the 'unprivileged' box.

 

[Davidoff]

hi,
if you see `unprivileged: 1` and your containers were not unprivileged previously, that might as well be the cause. you can convert your cts to privileged if you make a backup and restore after unchecking the 'unprivileged' box.

Thank you for the quick response Oguz. The containers were previously unprivileged. I did not do a backup and restore. I did an in-place upgrade following the steps of the upgrade guide and things appeared to go smoothly during that process.

Sorry, I should have also mentioned that I can see and read the mountpoints in containers just fine. When I run getfacl in the container, it shows the relevant user as having rwx rights on the mountpoint.

 

[oguz]

are the mappings still in the ct config files? is it configured correctly?

 

[Davidoff]

Thanks again Oguz. Yes, the mappings are still there. I did not change them either during or after the upgrade.

Just to see if perhaps the defaults had changed, I just now tried adding "acl=1" and "ro=0" to the mountpoint line in one of the containers, each individually and then both, restarting the container each time. No change - can't write within the container. And just to confirm, following is the mapping for one of the containers:

mp0: /vdata,mp=/vdata

 

[oguz]

i meant the uid mapping.

lxc.idmap = g 200 2000 1
or
lxc.idmap = g 2000 2000 1

like these

are they still in the container configs?

 

[Davidoff]

Thanks Oguz. Sorry just to clarify I abandoned the approach with uid mapping some time ago and instead use setfacl at the host level to assign rights to users within each container. I have no uid mappings at all for any of the containers.

Not sure if this is helpful but I tried manually starting a container using lxc-start to see if that would shed any light on the problem. I noticed two errors, one during startup of the container:

[  OK  ] Started Load Kernel Modules.
systemd-sysctl.service: Failed to reset devices.list: Operation not permitted
         Starting Apply Kernel Variables...
sys-kernel-config.mount: Failed to reset devices.list: Operation not permitted
         Mounting Kernel Configuration File System...
systemd-udevd.service: Failed to reset devices.list: Operation not permitted
         Starting udev Kernel Device Manager...
sys-kernel-config.mount: Mount process exited, code=exited status=32
sys-kernel-config.mount: Failed with result 'exit-code'.
[FAILED] Failed to mount Kernel Configuration File System.
See 'systemctl status sys-kernel-config.mount' for details.

And one during shutdown:

[FAILED] Failed unmounting /vdata.

Not sure if either is relevant to the issue I'm experiencing.

 

[Davidoff]

Hmmm. This gets curiouser and curiouser. I have another set of 2 drives set up as a ZFS RAID 1 zpool. In the host, it's mounted as /zdata. /zdata is bind mounted in one container in the same manner as /vdata. When I enter into that container, I can write just fine to /zdata, but not /vdata. getfacl shows the same rwx rights for both for root within the container. Very puzzling.

 

[Davidoff]

I tried something simple to try to narrow down what was causing the problem, so at the host level I ran:

chmod -R o+w /vdata

After doing so, I entered into a couple of containers and could write in /vdata in both. So the issue seems to be with ACL. I'd very much like to try to preserve what I've already implemented using ACLs, as I'd like to avoid allowing write rights for everyone to all bind-mounted directories.

 

[Davidoff]

I wasn't getting anywhere with ACLs so tried to revist the approach with lxc.idmap, basically following the approach set out in my first post above (and ensuring I corrected the typo for the group) and also following ObsidianBlk's advice and changing ownership to UID 2000 at the host level. I did this all for the host-level directory /vdata. For the chmod commands, I added -R to make it recursive. All the rest I did as set out in my first post.

This seemed to work and it seems that write rights are now available within each container, with one exception: There is a subdirectory under /vdata - /vdata/database - that has proven to be oddly stubborn. All the changes from the chgrp, chown and chmod commands did apply to it, and I've also checked to make sure that the ownership and rights correspond to 2000, container-data and rwx for owner and rws for group for both /vdata/database and all directories and files underneath it. I can also write to the directory at the host level. However, for some bizarre reason, I cannot write to that directory or any subdirectory under that directory in any container. I've also checked ownership and rights while in each container and they are the same.

Every other subdirectory under /vdata seems OK. It's just this one. The only way I seem to be able to write to that directory in any container is to chmod -R o+w at the host level, which I did just to test. I've checked the ACL settings which also give root in each container write rights to that directory, so I'm very puzzled by this.

Any thoughts on how I might be able to figure this one out would be most appreciated.

 

[Darkk]

I've been using Proxmox 5.4 for several months and recently upgraded to version 6 for home and work. Love the software. Each new version it gets better and better along with more ways to manage it via WebGUI vs command line. While nothing wrong working with command line but I find WebGUI easier to create clusters, CephFS and so on.

Now I've been playing around with LXC containers in hopes I can use it for Plex and shared local storage (XFS). Creating the container's storage mountpoint via WebGUI is pretty easy. However, I don't see an option to actually use the existing mountpoint on a local host storage without using command lines. Reason I am pointing this out is that WebGUI would have applied the correct permissions for the container to use the mountpoint and it's proper IDs for it. It's not a negative and this may get added in future upgrade.

So reading through the official Proxmox container guide I've use this to bind the mountpoint:​

​

Quote: "For example, to make the directory /mnt/bindmounts/shared accessible in the container with ID 100 under the path /shared, use a configuration line like mp0: /mnt/bindmounts/shared,mp=/shared in /etc/pve/lxc/100.conf. Alternatively, use pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared to achieve the same result."​

​

The command use pct set 100 -mp0 /mnt/bindmounts/shared,mp=/shared worked fine for me but not sure the proper way of setting the permissions for this shared folder. I don't think using chown / chgrp would be the proper way since it is managed by Proxmox.​

​

I am still getting write permission denied within the container. Any ideas?​

​

EDIT: For now I changed where I can share the folder and then do the pct set command. Basically created a shared folder in /home/root folder inside the container and then chmod -R o+w that shared folder on the host server. From then on I am able to write stuffs on the mounted folder inside the container.​

​

​

 

[ericgcollyer]

@Darkk, I was following you up until the point you said "chmod -R o+w that shared folder on the host server". Are you talking about running that command on the share you're going to access within the container? Or do you go on the host server, and run that command on the file within the container?

For example, let's say I have my SMB share mounted on /mnt/media on the host server. Do I run the chmod command on that folder (/mnt/media)? If so, doesn't that change the permissions of the files within the share? If this was not the case, let's say you're mounting the share at /media/share within the container. Did you go to the host machine and run the chmod command on the /media/share folder? If you did this, how did you accomplish this?

 

[carpler]

Sorry to 'revive' this old thread, but what Davidoff reported in the first post helped me solve the problem I had with an unprivileged container and the bind of a disk.
After much trial and error I was able to manage an HDD with ext4, but the procedure unfortunately doesn't work with a USB stick in FAT32: unfortunately this filesystem doesn't allow you to manage users and groups, so you can't apply commands such as chgrp or chown (it took me a long time and a lot of errors to understand that!).
In this regard I have two questions.
1) What alternative solution can there be for bind mount disks formatted in FAT32? (unimportant)
2) Could I have problems with NTFS disks? If so, does the same procedure apply as in the first post? (very important!)