Permission issues on rbind mounts to LXC

yarcod

Member
Sep 30, 2020
17
1
8
36
Hi

I'm trying to setup my new environment, but have ran into some permission issues. I thought myself quite well-versed in the permissions' department by now, given all the issues I've had to solve before, but this one has left be dumbfounded.

On the host side I have several ZFS datasets that I would like to mount to a few **unprivileged** LXCs. Currently the approach is to set the group permissions on the mounts to a `container-data` group, with GID 101000. This group is mapped in `/etc/subgid` as 1000. The group is set to have full permissions (rwx) on the host. When I browse the files already present on the drive, I can see that the group indeed is set to have the right permissions, i.e.
Code:
root@proxmox:~# ls -lash /Tank2/Downloads/
 17K drwxrws---+   6 root container-data   6 Nov  3  2020 .
 17K drwxrws---+   6 root container-data   6 Aug 12 10:23 ..
 41K drwxrwxr-x+ 113 _apt container-data 120 Apr  1 20:59 completed
 33K drwxrwxr-x+   2 _apt container-data   2 Apr  1 20:59 incomplete
The mount is added directly to `/etc/pve/lxc/<id>.conf` as:
lxc.mount.entry: /Tank2/Downloads storage/media none rbind,create=dir,optional 0 0

At this point, I expected to be able to list the content of the mount `/storage/media' and add new files. However, I am only able to access the file system but not list the contents already present inside. Not the it throws me an error, the files/directories just won't show up.

If I create a file from the container as a user part of the `container-data` group (they are called the same on both host and guest), I get permission denied. But creating a file as root from the container works, but that file is only visible/present when viewing from the container. Browsing from the host won't show that new file.

Another bind (`rbind`) mount behaves similarly, but if I instead spin up a Docker container in the LXC, which in turn creates a file from within it will have written the files with the correct permissions on both ends. (While writing this, I realise that this could be because the files are created as root in the Docker container and then having their permissions changed after the fact...). Those files are persisting just fine to the host. But, from what I have setup, shouldn't only the user in the `container-data` group have write permissions? And what could I have done to make a bind mount appear to be working but no data is actually being stored properly. And where do my testfiles end up if not on the ZFS datasets?

So many questions! Any tips, hints, ideas or guesses are welcome! Thanks in advance!
 
Last edited:
And where do my testfiles end up if not on the ZFS datasets?
I guess in the root filesystem of your LXC like it normally happens when you write to a mountpoint that isn't mounted?

And are you sure you are not missing a "/" infront of your "storage/media"? Maybe you write to "/storage/media" inside the LXC but the dataset is bind-mounted somewhere else.
 
I guess in the root filesystem of your LXC like it normally happens when you write to a mountpoint that isn't mounted?
Nope, can't find it anywhere. When the mountpoint is removed, however, the files in /storage/media remains. So I guess the mountpoint never was attached correctly in the first place?

And are you sure you are not missing a "/" infront of your "storage/media"? Maybe you write to "/storage/media" inside the LXC but the dataset is bind-mounted somewhere else.
I've tried both with and without '/'. The only difference I noticed was when I initially attempted to mount it, I got a (correct) permission's error only when I removed the '/'; if I had the '/' it would just tell me nothing and just happily restart the container without the mount. Don't know if that makes a difference in the end.

However, after doing some additional digging locally I can add a few more, (to me) confusing facts:
  • When I remove the other line (/storage/appdata), the mountpoint in the container still contain all the data. I expected it to be empty when the mount was removed?
  • If I remove the problematic config line during a run, remove the resulting mount in the CT, re-enable the line again and restart, the container will not create /storage/media again. I guess that indicates that something is amiss...?
    • If I, however, re-enable it without the leading '/' on the CT mount path it will be re-created inside the container. Still empty though...
  • I also still do not get the permissions in this case -- the CT root user is mapped to UID 10000 in Proxmox, and the other user ('default-user') is mapped to 101000. But despite the permissions on the host being set to only allow writes from 'default-user', root is able to write to the mountpath. Permissions are set both with directory permissions as well as ACLs (I don't need ACLs per se, I have just been trying to get it to work) [/list}
 
Nope, can't find it anywhere. When the mountpoint is removed, however, the files in /storage/media remains. So I guess the mountpoint never was attached correctly in the first place?
Jep. Sounds like it never was mounted so you wrote stuff to the root filesystem instead.
 
Jep. Sounds like it never was mounted so you wrote stuff to the root filesystem instead.
Oh, sorry, I misinterpreted your comment; I thought you wanted to tell me that it ended up in /.

What can I do to troubleshoot this issue though? I can't see anything in /var/log/messages that indicates where the issue is. Something is going wrong in the mounting process, but I can't find anything telling me why.
 
I did, and it just shows one of the mounts. Attaching just the problematic one does not show up in either command.

However, I tried to just attach the root /Tank2 instead of the sub-dataset and that works. When then adding the other mountpoint, /storage/appdata, that one shows up as well. So now I almost am where I want, but not entirely satisfied with the solution or understanding the underlying problem. Why can one be attached and the other not? Why can't I find any errors when it tries to mount it?

One thing that I haven't brought up, because I didn't think it relevant, is that both /storage/appdata and the problematic one, /Tank2/Downloads, are both on the same base ZFS dataset, called Tank2. Does it matter to lxc or (r)bind that they are located on the same dataset but mounted on totally different locations on the host? (The reason for having different moutpoints is purely legacy and laziness, and, again, I didn't that it could matter).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!