Containers fail with "safe_mount_beneath_at: 1106 Function not implemented"

N

neilb

Member
Jul 24, 2017
2
0
6
63
Proxmox 6.3.6

Following a reboot, all VMs re-started without problem but all containers failed.

Initially error was zfs complaining that mount points were missing. zfs list confirmed.

Code: 
root@pve:~# zfs list
NAME                      USED  AVAIL     REFER  MOUNTPOINT
rpool                    3.24T   836G     1.19G  none
rpool/ROOT               3.23T   836G       96K  none
rpool/ROOT/pve-1         3.23T   836G     3.23T  /
rpool/data                 96K   836G       96K  none
rpool/subvol-100-disk-1    96K   836G       96K  none
rpool/subvol-101-disk-1    96K  8.00G       96K  none
rpool/subvol-102-disk-0    96K   836G       96K  none
rpool/subvol-103-disk-1    96K  8.00G       96K  none
rpool/subvol-105-disk-1    96K  30.0G       96K  none
rpool/subvol-106-disk-1    96K  8.00G       96K  none
...

And...
Code: 
root@pve:~# zfs get mountpoint rpool
NAME   PROPERTY    VALUE       SOURCE
rpool  mountpoint  none        local

So...
Code: 
root@pve:~# zfs set mountpoint=/rpool rpool

After which.....
Code: 
root@pve:~# zfs list
NAME                      USED  AVAIL     REFER  MOUNTPOINT
rpool                    3.24T   836G     1.19G  /rpool
rpool/ROOT               3.23T   836G       96K  none
rpool/ROOT/pve-1         3.23T   836G     3.23T  /
rpool/data                 96K   836G       96K  /rpool/data
rpool/subvol-100-disk-1    96K   836G       96K  /rpool/subvol-100-disk-1
rpool/subvol-101-disk-1    96K  8.00G       96K  /rpool/subvol-101-disk-1
rpool/subvol-102-disk-0    96K   836G       96K  /rpool/subvol-102-disk-0
rpool/subvol-103-disk-1   160K  8.00G       96K  /rpool/subvol-103-disk-1
rpool/subvol-105-disk-1    96K  30.0G       96K  /rpool/subvol-105-disk-1
rpool/subvol-106-disk-1   160K  8.00G       96K  /rpool/subvol-106-disk-1
...

All good. But now the containers fail to boot with the error...
Code: 
root@pve:~# pct start 103
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 51(dev)
__sync_wait: 36 An error occurred in another process (expected sequence number 7)
__lxc_start: 1999 Failed to spawn container "103"
startup for container '103' failed

Similarly, if I clone a container (to ensure a new mountpoint) the new container fails to boot with the same error.

Grateful for thoughts on solving this.

Thanks
 
The debug log suggests that the rootfs is not mounted as /etc cannot be found in the CT.

lxc-start 103 20210421072817.742 INFO lsm - lsm/lsm.c:lsm_init:40 - Initialized LSM security driver AppArmor
lxc-start 103 20210421072817.742 INFO conf - conf.c:run_script_argv:331 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "103", config section "lxc"
lxc-start 103 20210421072818.225 DEBUG conf - conf.c:run_buffer:303 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 103 lxc pre-start produced output: /etc/os-release file not found and autodetection failed, falling back to 'unmanaged'
WARNING: /etc not present in CT, is the rootfs mounted?
got unexpected ostype (unmanaged != ubuntu)
lxc-start 103 20210421072818.234 DEBUG terminal - terminal.c:lxc_terminal_peer_default:672 - Using terminal "/dev/tty" as proxy
lxc-start 103 20210421072818.234 DEBUG terminal - terminal.c:lxc_terminal_winsz:61 - Set window size to 110 columns and 28 rows
lxc-start 103 20210421072818.257 DEBUG seccomp - seccomp.c:parse_config_v2:656 - Host native arch is [3221225534]
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "[all]"
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:566 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:566 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:566 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
lxc-start 103 20210421072818.257 INFO seccomp - seccomp.c:do_resolve_add_rule:566 - Adding native rule for syscall[304:eek:pen_by_handle_at] action[327681:errno] arch[0]
....
(truncated)

Tried mounting the container with pct mount 103. The mountpoint /var/lib/lxc/103/rootfs/ has dev proc and sys directories only...in particular, I believe there should be an empty run directory. However, it is not created.

Assistance still required.

Thanks
 
Last edited:
Hi,

seems that I have a similar problem. I have that error after updating Proxmox and Container. But I don't use ZFS. I can mount the container but it will not start. I posted about that in the german subforum,

Code: 
lxc-start 101 20210423085537.986 ERROR    utils - utils.c:__safe_mount_beneath_at:1106 - Function not implemented - Failed to open 54(dev)

I can mount the container and that is what I see, also the run folder:
Code: 
root@pve:~# pct mount 101
mounted CT 101 in '/var/lib/lxc/101/rootfs'
root@pve:~# ls -l /var/lib/lxc/101/rootfs
total 92
drwxr-xr-x  2 root root  4096 Apr 19 08:45 bin
drwxr-xr-x  2 root root  4096 May 13  2019 boot
drwxr-xr-x  2 root root  4096 Dec 10  2008 dev
drwxr-xr-x 72 root root  4096 Apr 23 15:21 etc
-rw-r--r--  1 root root     0 Apr 23 15:21 fastboot
drwxr-xr-x  2 root root  4096 May 13  2019 home
drwxr-xr-x 10 root root  4096 Jul  8  2019 lib
drwxr-xr-x  2 root root  4096 Jul  8  2019 lib64
drwx------  2 root root 16384 Sep  4  2019 lost+found
drwxr-xr-x  2 root root  4096 Jul  8  2019 media
drwxr-xr-x  3 root root  4096 Sep 14  2019 mnt
drwxr-xr-x  3 root root  4096 Sep 14  2019 opt
drwxr-xr-x  2 root root  4096 May 13  2019 proc
drwx------  8 root root  4096 Apr 19 08:48 root
drwxr-xr-x  3 root root  4096 Jul  8  2019 run
drwxr-xr-x  2 root root  4096 Apr 19 08:45 sbin
drwxr-xr-x  2 root root  4096 Jul  8  2019 srv
drwxr-xr-x  2 root root  4096 May 13  2019 sys
drwxrwxrwt  7 root root  4096 Apr 19 08:45 tmp
drwxr-xr-x 10 root root  4096 Jul  8  2019 usr
drwxr-xr-x 11 root root  4096 Jul  8  2019 var

Code: 
root@pve:~# ls -l /var/lib/lxc/101/rootfs/run/
total 4
drwxrwxrwt 2 root root 4096 Jul  8  2019 lock
-rw-rw-r-- 1 root utmp    0 Jul  8  2019 utmp

BR
Mike
 
You must log in or register to reply here.
Top Bottom