Container with mp0 configuration won't start via GUI, OK via `lxc-start`

[CDuv]

I've followed the Mounting NFS share to an unprivileged LXC tutorial to mount a NFS share in a container.

The mount on Proxmox node works (I can read/write /mnt/nfsshare) but as soon as I add the mp0: /mnt/nfsshare/,mp=/foo line to the LXC configuration file (/etc/pve/lxc/100.conf) the container won't start anymore, from the GUI:

run_buffer: 571 Script exited with status 13
lxc_init: 845 Failed to run lxc.hook.pre-start for container "100"
__lxc_start: 2047 Failed to initialize container "100"
TASK ERROR: startup for container '100' failed

I've started the container using lxc-start -n 100 -F -lDEBUG -o lxc-100.log for more helpful messages but it started just fine:

OpenRC 0.62.3 is starting up Linux 6.17.2-1-pve (x86_64) [LXC]

* /proc is already mounted
* /run/lock: creating directory
* /run/lock: correcting owner
* Caching service dependencies ... [ ok ]
* Mounting local filesystems ... [ ok ]
* Creating user login records ... [ ok ]
* Cleaning /tmp directory ... [ ok ]
* Remounting devtmpfs on /dev ...mount: mounting dev on /dev failed: Permission denied
[ !! ]
* Mounting /dev/mqueue ... [ ok ]
* Starting networking ... * lo ... [ ok ]
* eth0 ... [ ok ]
* Starting busybox syslog ... [ ok ]
* Starting busybox crond ... [ ok ]

Welcome to Alpine Linux 3.22
Kernel 6.17.2-1-pve on x86_64 (/dev/console)

acmecontainer login:

And GUI can access LXC's shell
A simpler lxc-start -n 100 also works.

Removing the mp0: /mnt/nfsshare/,mp=/foo allows starting the LXC from the GUI.

What's the issue there?

Here is LXC's configuration:

arch: amd64
cores: 1
hostname: acmecontainer
memory: 512
mp0: /mnt/nfsshare/,mp=/foo
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.0.1,hwaddr=BC:24:11:ff:ff:ff,ip=192.168.0.2/24,type=veth
onboot: 1
ostype: alpine
rootfs: local-lvm:vm-100-disk-0,size=2G
swap: 512
tags: test
unprivileged: 1

I am using PVE 9.1.1 ("pve-manager/9.1.1/42db4a6cf33dac83 (running kernel: 6.17.2-1-pve)")

 

[Impact]

What's this say?

pct start 100 --debug

 

[CDuv]

It gets a "Permission denied" for the (PVE side) mountpoint with a leading "/":

run_buffer: 571 Script exited with status 13
lxc_init: 845 Failed to run lxc.hook.pre-start for container "100"
__lxc_start: 2047 Failed to initialize container "100"
0 hostid 100000 range 65536
INFO lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: cannot open directory //mnt/nfsshare: Permission denied

ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 13
ERROR start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "100"
ERROR start - ../src/lxc/start.c:__lxc_start:2047 - Failed to initialize container "100"
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
startup for container '100' failed

 

[Impact]

Hmm. I don't use NFS myself so I can't help with specifics related to it but this would be useful to see

findmnt -T /mnt/nfsshare/
namei -l /mnt/nfsshare/

 

[CDuv]

When unmounted:

root@pve:~# findmnt -T /mnt/nfsshare/
TARGET SOURCE               FSTYPE OPTIONS
/      /dev/mapper/pve-root ext4   rw,relatime,errors=remount-ro

root@pve:~# namei -l /mnt/nfsshare/
f: /mnt/nfsshare/
drwxr-xr-x root root /
drwxr-xr-x root root mnt
drwxr-xr-x root root nfsshare

When mounted:

root@pve:~# findmnt -T /mnt/nfsshare/
TARGET        SOURCE                FSTYPE OPTIONS
/mnt/nfsshare systemd-1             autofs rw,relatime,fd=85,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1131652
/mnt/nfsshare 192.168.0.3:/mnt/data nfs4   rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.2,local_lock=none,addr=192.168.0.3

root@pve:~# namei -l /mnt/nfsshare/
f: /mnt/nfsshare/
drwxr-xr-x root root /
drwxr-xr-x root root mnt
drwxrwx--- root root nfsshare

 

[fabian]

drwxrwx--- root root nfsshare

this means it's not accessible for unprivileged users..

 

[CDuv]

So this is something pct will check/stumble on but not lxc-start?

According to ls, mount point has extended permissions / ACLs

root@pve:~#ls -ld /mnt/nfsshare
drwxrwx---+ 5 root root 5 Dec 26 03:09 /mnt/nfsshare

So I've installed the acl package to execute getfacl but don't see any special permission being set:

root@pve:~# getfacl /mnt/nfsshare
getfacl: Removing leading '/' from absolute path names
# file: mnt/nfsshare
# owner: root
# group: root
user::rwx
group::rwx
other::---

When unmounted:

root@pve:~# getfacl /mnt/nfsshare
getfacl: Removing leading '/' from absolute path names
# file: mnt/nfsshare
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

NFS server (192.168.0.3) is a TrueNAS server.
I've updated the dataset permissions to give "everyone@" the "Allow | Modify" permission and now the container starts fine from both the GUI and pct start 100.
I am not sure if it's the best thing to do, but as it's only a test I'd say it's OK (I'll setup NFS credentials with Kerberos later).

But, instead of changing the permissions on NFS server could I have added ACL to the mountpoint or adapt my mount unit ("/etc/systemd/system/mnt-nfsshare.mount" see below)?

[Unit]
Description=nfsshare
After=network.target

[Mount]
What=192.168.0.3:/mnt/data
Where=/mnt/nfsshare
Type=nfs
Options=_netdev,auto

[Install]
WantedBy=multi-user.target

With your help it now looks obvious the container being unprivileged was an issue for accessing a path with o= permission.