Understanding lxc.idmap, /etc/sub{u,g}id, cannot login as root, root lost permissions?

virtManager

Member
Jun 11, 2020
28
4
8
44
Hi,

I'm a proxmox-noob, been messing around with lxc.idmap, /etc/sub{u,g}id etc and I finally got something working but I cannot login as root and when I "cheat" using "pct enter 101" it's like the root user lost his root permissions so this is really weird and I've been googling around, trying to understand this problem. Instead I'll just describe what I see and have done:
Code:
# pct config 101
...
mp0: /mnt/test_share,mp=/mnt/test_share
...
unprivileged: 1
lxc.idmap: u 0 1000 1
lxc.idmap: g 0 1000 1
lxc.idmap: u 1 100001 65535
lxc.idmap: g 1 100001 65535

So I wanted that root-owned files in the bind mount appeared, not as uid/gid 100000, but as pve : pve, which exists on the host. I'll later setup a network share for this bind-mounted folder, from within the container (CIFS/NFS or maybe FTP, haven't completely decided yet). But I want root:root (container) --> pve : pve (host), where uid/gid for pve is 1000:1000. Continuing from here I have just added two extra lines to each of these files, telling that I want to permit mapping from container to uid/gid 1000:1000:
Code:
root@proxmox:/etc/pve/lxc# cat /etc/subuid
root:100000:65536
pve:165536:65536
root:1000:1
pve:1000:1

Code:
root@proxmox:/etc/pve/lxc# cat /etc/subgid
root:100000:65536
pve:165536:65536
root:1000:1
pve:1000:1

Luckily I could google for figuring out this configuration but for a long time I thought the configuration was wrong: The weird thing is that when I made these changes, restarted the container, I could no longer login as root using the GUI Console... I thought I had made a mistake - until I discovered that I could type "pct enter 101" or "lxc-attach 101" and get access. Then I could "touch somefile", exit the container and verify that the file that was owned by root:root in the container, while that same file(s) on the host appeared to be not 100000:100000, but pve : pve. So far so good...

I was - and am - still wondering about what happened to the root-user and why I cannot login as root anylonger to the container using the GUI Console. Inside the "pct enter 101"-terminal, I first verified that I in fact was/is root - next I tried to add a new user "hostuser" - but I also cannot do that as root:
Code:
/ # whoami
root

/ # adduser hostuser
adduser: /etc/passwd: Permission denied

When I checked here: https://wiki.alpinelinux.org/wiki/Setting_up_a_new_user - I realized that this shouldn't happen. It's supposed to accept that command, create a new user and ask for that users password. And when I saw this - that root cannot even make a user, I was thinking, hmm, that seems to be related with the fact that I cannot login as root. But how? I don't understand that?

I thought the mapping was not about changing root permissions "inside" the container, I thought the mapping would only affect something outside the container, i.e. something in the host/container-communication? If/when I remove the id-mapping stuff, the root user works as normal or before. But that doesn't solve the original problem, because I want the mapping to happen, while root should function as normally.

There is something I don't understand about this - and this is where I thought, I have to create this post and hopefully someone is willing to explain the missing part of this... I tried googling and reading, but couldn't see exactly this problem described anywhere. At the same time I'm not happy with the situation (either the mapping isn't as I want or the root-account is screwed up) - so I would be very grateful if anyone could help with ideas or explain what I'm missing, thanks!
 
Last edited:
UPDATE: It's *NOT* working (yesterday I wrote things were working, I've completely modified that=this post now, as I thought I haven't changed my /etc/pve/lxc/101.conf but apparently I did and forgot and thought things worked. So here's the deal: I am running alpine linux 3.12 and didn't ran the "setup-alpine"-script from the beginning. But I don't think it makes a difference, also I don't think it makes a difference that I've messed some hours with setting up ftpd, smb-server and nfs-server (didn't work), installed packages needed for these. I found a post, asking to verify if /etc/shadow contains a line for the root user, that's not starting with "root:*:". In that process I discovered some interesting things and error messages and I'll just also provide the full config-file here:

Code:
root@proxmox:/etc/pve/lxc# cat 101.conf
arch: amd64
cores: 2
hostname: alpine-3.12
memory: 1024
mp0: /mnt/test_share,mp=/mnt/test_share
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=32:XX:YY:ZZ:A7:7D,ip=dhcp,ip6=dhcp,type=veth
ostype: alpine
rootfs: local:101/vm-101-disk-0.raw,size=8G
swap: 0
unprivileged: 1
lxc.idmap: u 0 1000 1
lxc.idmap: g 0 1000 1
lxc.idmap: u 1 100001 65535
lxc.idmap: g 1 100001 65535

When idmap is in use, i.e. when the last 4 "lxc.idmap"-lines appended to my /etc/pve/lxc/101.conf-file, things aren't working at all. I can however use the "pct enter 101"-trick, see here:
Code:
/ # whoami
root

/ # ls -latrh /etc/shadow
-rw-r-----    1 nobody   shadow       717 Sep 27 07:09 /etc/shadow

/ # grep -in root /etc/shadow
grep: /etc/shadow: Permission denied

/ # cat /etc/shadow
cat: can't open '/etc/shadow': Permission denied
When idmap is *NOT* in use, i.e. when the above-mentioned 4 lines are commented out with a '#'-sign (in the /etc/pve/lxc/101.conf-file shown in the top), I don't need to use the "pct enter 101"-trick, I can just login as root user via the webUI/GUI Console and notice that where /etc/shadow was owned by "nobody" before (which is incorrect), it's not correctly owned by root and I can examine the contents:
Code:
alpine-3:~# ls -latrh /etc/shadow
-rw-r-----    1 root     shadow       717 Sep 27 07:09 /etc/shadow

alpine-3:~# cat /etc/shadow
root:$6$SfbBnOy1RRVoguDT$ez..............TVdAzTJfgrST9/MJdU0:18532:0:::::
...

Let me show something else: If I don't have the 4 idmap-lines, this is what I get:
Code:
# lxc-start -F -n 101 -l DEBUG -o /tmp/101.log

   OpenRC 0.42.1.9028084776 is starting up Linux 5.4.60-1-pve (x86_64) [LXC]

 * /proc is already mounted
 * /run/openrc: creating directory
 * /run/lock: creating directory
 * /run/lock: correcting owner
 * Caching service dependencies ... [ ok ]
 * Creating user login records ... [ ok ]
 * Wiping /tmp directory ... [ ok ]
 * Starting networking ... *   lo ...ip: RTNETLINK answers: File exists
 [ !! ]
 *   eth0 ...udhcpc: started, v1.31.1
udhcpc: sending discover
udhcpc: sending discover
udhcpc: sending select for 192.168.1.157
udhcpc: lease of 192.168.1.157 obtained, lease time 86400
 [ ok ]
 * Starting busybox syslog ... [ ok ]
 * Starting busybox crond ... [ ok ]
 * Starting sshd ... [ ok ]

Welcome to Alpine Linux 3.12
Kernel 5.4.60-1-pve on an x86_64 (/dev/console)

This is nice, right? When I add the 4 lines, this is what I get:
Code:
root@proxmox:/etc/pve/lxc# lxc-start -F -n 101 -l DEBUG -o /tmp/101.log                       

   OpenRC 0.42.1.9028084776 is starting up Linux 5.4.60-1-pve (x86_64) [LXC]
                                                                        
 * /proc is already mounted                                                                                                                                               
 * /run/openrc: creating directory                                                                                                                                         
 * /run/lock: creating directory                                                                                                                                           
 * /run/lock: correcting owner
 * Caching service dependencies ... [ ok ]
 * Creating user login records ... [ ok ]     
 * Wiping /tmp directory ...rm: can't remove './.ICE-unix': Operation not permitted
rm: can't remove './.X11-unix': Operation not permitted                                                                                                                   
 [ ok ]                                                       
 rm: can't remove '/tmp/.ICE-unix': Operation not permitted
rm: can't remove '/tmp/.X11-unix': Operation not permitted
chmod: /tmp/.ICE-unix: Operation not permitted
chmod: /tmp/.X11-unix: Operation not permitted
 * Starting networking ... *   lo ...ip: RTNETLINK answers: File exists
 [ !! ]
 *   eth0 ...udhcpc: started, v1.31.1
udhcpc: sending discover                       
udhcpc: sending discover
udhcpc: sending select for 192.168.1.157     
udhcpc: lease of 192.168.1.157 obtained, lease time 86400           
/usr/share/udhcpc/default.script: line 93: can't create /etc/resolv.conf.264: Permission denied                                                                           
/usr/share/udhcpc/default.script: line 97: can't create /etc/resolv.conf.264: Permission denied                                                                           
/usr/share/udhcpc/default.script: line 100: can't create /etc/resolv.conf.264: Permission denied                                                                           
chmod: /etc/resolv.conf.264: No such file or directory
mv: can't rename '/etc/resolv.conf.264': No such file or directory
 [ ok ]                             
 * Starting busybox syslog ...Sep 28 23:13:55 alpine-3 syslog.info syslogd started: BusyBox v1.31.1                                                                       
 [ ok ]                     
Sep 28 23:13:55 alpine-3 daemon.info init: starting pid 316, tty '': '/sbin/openrc default'                                                                               
 * Starting busybox crond ...Sep 28 23:13:55 alpine-3 cron.info crond[336]: crond (busybox 1.31.1) started, log level 8                                                   
Sep 28 23:13:55 alpine-3 cron.err crond[336]: root: Permission denied
 [ ok ]                                                               
sshd: no hostkeys available -- exiting.
 * ERROR: sshd failed to start                                             
Sep 28 23:13:55 alpine-3 daemon.err /etc/init.d/sshd[343]: ERROR: sshd failed to start
Sep 28 23:13:55 alpine-3 daemon.info init: starting pid 361, tty '': '/sbin/getty 38400 console'                                                                           
Sep 28 23:13:55 alpine-3 daemon.info init: starting pid 362, tty '/dev/tty1': '/sbin/getty 38400 tty1'                                                                     
Sep 28 23:13:55 alpine-3 daemon.info init: starting pid 363, tty '/dev/tty2': '/sbin/getty 38400 tty2'                                                                     
                      
Welcome to Alpine Linux 3.12             
Kernel 5.4.60-1-pve on an x86_64 (/dev/console)
                                                                          
alpine-3.12 login: Sep 28 23:14:00 alpine-3 cron.err crond[336]: root: Permission denied

Not very nice, eh? As I'm still learning proxmox, I unfortunately only just learned these things, ideally I should've provided this info from the beginning... I hope these additional details can help someone lead me in the right way of understanding what's wrong with my idmap-config - or should I just not idmap uid/gid 0? I had/have the impression that all user accounts incl root can be idmapped...?

Again, I thought the idmap was only supposed to affect something "externally" to the container. But it really causes severe problems, inside the container - I don't understand why. If anyone can shed light on this, come up with a meaningful explanation (or some additional insights/help), I would be very grateful, thanks.
 
Last edited:
Just curious if you ever figured this out. I was trying to do the same thing and had the same problems, but with nextcloud turnkey. I was just trying to get root to map as the UID 1000 user on the host and haven't figured it out. I'm still learning about these, but maybe you had the magic bullet I was missing. I cannot log in or access when I change the root mapping to a different UID/GID.
 
Hi, no, sorry I never completely figured this stuff out - except I think I did a workaround with letting the container be a privileged container - as I vaguely remember it. I'm not really using that machine anymore but will soon setup a new machine, so I you figure this stuff out, please share! I'm also surprised that no-one seemed to explain it so I could understand it - or maybe I'm just too stupid to understand it. In any case, I think I found a bad solution that worked and lived with that for some time. Unfortunately I cannot remember the details and don't have access to this pc anymore. I'm happy that I'm not the only one in this boat - it must mean I'm not so stupid after all :)
 
Usually your LXCs root, with UID 0 inside the LXC, should be UID 100000 on the host. If you then map UID 0 inside the LXC to UID 1000 on the host all the existing files will still be owned by UID 100000. If you then want to access them from inside the LXC you got no rights to do this, as you are accessing files owned by UID 100000 with UID 1000.
So you need to login as root into your host and then chown every file and folder of that LXC to UID 1000 that is owned by UID 100000.
 
Usually your LXCs root, with UID 0 inside the LXC, should be UID 100000 on the host. If you then map UID 0 inside the LXC to UID 1000 on the host all the existing files will still be owned by UID 100000. If you then want to access them from inside the LXC you got no rights to do this, as you are accessing files owned by UID 100000 with UID 1000.
So you need to login as root into your host and then chown every file and folder of that LXC to UID 1000 that is owned by UID 100000.
So glad i found this post! finally can login to root in my LXC with Root in CT mapped to 1001 in host, i do have a problem tho.

after login i ran apt update, apt upgrade and got this error:

Code:
/usr/bin/mandb: can't chmod /var/cache/man/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: fopen /var/cache/man/zh_CN/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/de/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pt_BR/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sv/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pl/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/nl/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fi/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/da/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ro/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ko/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sr/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pt/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/hu/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/tr/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ru/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/id/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/cs/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/it/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ja/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sl/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/es/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fr/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/zh_TW/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/uk/2840: Permission denied

i changed permissions of the LXC by:
Code:
pct mount 109

chown -R 1001:1001 /var/lib/lxc/109/rootfs

Im guessing some permissions are screwed doing it this way.

Is there possibly a way to create an LXC with 1001 permissions instead of 100000?

thanks for any help
 
So glad i found this post! finally can login to root in my LXC with Root in CT mapped to 1001 in host, i do have a problem tho.

after login i ran apt update, apt upgrade and got this error:

Code:
/usr/bin/mandb: can't chmod /var/cache/man/CACHEDIR.TAG: Operation not permitted
/usr/bin/mandb: fopen /var/cache/man/zh_CN/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/de/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pt_BR/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sv/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pl/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/nl/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fi/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/da/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ro/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ko/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sr/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/pt/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/hu/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/tr/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ru/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/id/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/cs/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/it/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/ja/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/sl/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/es/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/fr/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/zh_TW/2840: Permission denied
/usr/bin/mandb: fopen /var/cache/man/uk/2840: Permission denied

i changed permissions of the LXC by:
Code:
pct mount 109

chown -R 1001:1001 /var/lib/lxc/109/rootfs

Im guessing some permissions are screwed doing it this way.

Is there possibly a way to create an LXC with 1001 permissions instead of 100000?

thanks for any help

I figured it out. It only took a few hours, 10 broken LXC containers, accidentally chowning important files on the host system, and pulling out some hair. This is what did it for me:

In this example, I will use container ID 202.
I want to map root: user 0 group 0 (container) -> shinobi: user 1500 group 1500 (host).

After changing the appropriate configuration files: /etc/pve/lxc/202.conf, /etc/subuid and /etc/subgid, I still had to chown the files in the container itself.

This is the important part of my /etc/pve/lxc/202.conf:

Code:
unprivileged: 1
lxc.idmap: u 0 1500 1
lxc.idmap: g 0 1500 1
lxc.idmap: u 1 100001 65535
lxc.idmap: g 1 100001 65535

1. First, mount the container: pct mount 202
2. Chown all files previously owned by root, for me it was uid 10000:

Make sure to use chown -h when targeting symlinks so you don't mess up your host like I did.

Code:
find /var/lib/lxc/202/rootfs -user 100000 -type f -exec chown shinobi:shinobi {} +
find /var/lib/lxc/202/rootfs -user 100000 -type d -exec chown shinobi:shinobi {} +
find /var/lib/lxc/202/rootfs -user 100000 -type l -exec chown -h shinobi:shinobi {} +

3. You *also* need to chown files whose previous group was root, but where the user was not root. For example, /var/lib/apt/lists/partial has user _apt but group root. This will only override the group.

Code:
find /var/lib/lxc/202/rootfs -group 100000 -type f -exec chown :shinobi {} +
find /var/lib/lxc/202/rootfs -group 100000 -type d -exec chown :shinobi {} +
find /var/lib/lxc/202/rootfs -group 100000 -type l -exec chown -h :shinobi {} +

4. Don't forget to unmount with pct unmount 202

Maybe it's too late for you, but hopefully, this helps someone.
 
Last edited:
This is what did it for me:

In this example, I will use container ID 202.
I want to map root: user 0 group 0 (container) -> shinobi: user 1500 group 1500 (host).

After changing the appropriate configuration files: /etc/pve/lxc/202.conf, /etc/subuid and /etc/subgid, I still had to chown the files in the container itself.

This is the important part of my /etc/pve/lxc/202.conf:

Code:
unprivileged: 1
lxc.idmap: u 0 1500 1
lxc.idmap: g 0 1500 1
lxc.idmap: u 1 100001 65535
lxc.idmap: g 1 100001 65535

1. First, mount the container: pct mount 202
2. Chown all files previously owned by root, for me it was uid 10000:

Make sure to use chown -h when targeting symlinks so you don't mess up your host like I did.

Code:
find /var/lib/lxc/202/rootfs -user 100000 -type f -exec chown shinobi:shinobi {} +
find /var/lib/lxc/202/rootfs -user 100000 -type d -exec chown shinobi:shinobi {} +
find /var/lib/lxc/202/rootfs -user 100000 -type l -exec chown -h shinobi:shinobi {} +

3. You *also* need to chown files whose previous group was root, but where the user was not root. For example, /var/lib/apt/lists/partial has user _apt but group root. This will only override the group.

Code:
find /var/lib/lxc/202/rootfs -group 100000 -type f -exec chown :shinobi {} +
find /var/lib/lxc/202/rootfs -group 100000 -type d -exec chown :shinobi {} +
find /var/lib/lxc/202/rootfs -group 100000 -type l -exec chown -h :shinobi {} +

4. Don't forget to unmount with pct unmount 202

Maybe it's too late for you, but hopefully, this helps someone.
I can confirm that this solution has worked for me. It fixed the login issue and other weirdness after following the guide https://pve.proxmox.com/wiki/Unprivileged_LXC_containers to mount my SMB share from Unraid to my LXC Container on Proxmox with the root UID "0" in lxc.idmap = u 0 0 1 / lxc.idmap = g 0 0 1

(Hope this makes this post more google-able)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!