LXC not starting anymore

[Alex2801]

Hi together
Today, suddenly one of my containers wouldn't start anymore.

The following error appears if i start it:

 #pct start 105
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 51(dev)
__sync_wait: 36 An error occurred in another process (expected sequence number 7)
startup for container '105' failed

the container looks like (hostname/domain/ip/mac changed):

#pct config 105
arch: amd64
cores: 4
hostname: test.domain.de
memory: 2048
mp0: dataHDD0:subvol-105-disk-1,mp=/mnt/ncdata,backup=1,size=3000G
mp1: dataHDD0:subvol-105-disk-9,mp=/mnt/backuptemp,size=100G
nameserver: 10.10.10.254
net0: name=eth0,bridge=vmbr0,firewall=1,gw=10.10.10.254,hwaddr=C2:33:03:2B:BF:AE,ip=10.10.10.11/24,tag=10,type=veth
ostype: ubuntu
rootfs: dataHDD0:subvol-105-disk-0,size=30G
searchdomain: domain.de
swap: 2048

zfs mounts seem to be there:

dataHDD0                                                 mounted   yes      -
dataHDD0/encr                                            mounted   yes      -
dataHDD0/encr/subvol-105-disk-0                          mounted   yes      -
dataHDD0/encr/subvol-105-disk-1                          mounted   yes      -
dataHDD0/encr/subvol-105-disk-9                          mounted   yes      -

The debug log of the lxc startup is attached.

I have tried to search around but i did not found something that appears to match the error.
Can someone give me a hint how i can debug this further or how i can solve the problem?


Thanks!
Alex

 

[leesteken]

In your attached log, I see:
lxc-start 105 20210412170228.192 NOTICE start - start.c:start:2087 - Exec'ing "/sbin/init" lxc-start 105 20210412170228.192 ERROR start - start.c:start:2090 - No such file or directory - Failed to exec "/sbin/init"
I have seen something like this once before when I accidentally and unknowingly did not backup the system directories such as /sbin and did a restore of the container backup overwriting the container with a incomplete filesystem. Can you check whether subvol-105-disk-0 actually contains /sbin/init ?

 

[BlackiE]

Hi, suddenly also i have the same problem today. All my unprivileged containers won't start any more. I noticed that privileged containers will start. I updated and restarted my server as homelab. I tried to start a new unprivileged container;

# pct start 102
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 51(dev)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(full)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(null)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(random)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(tty)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(urandom)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(zero)
lxc_pivot_root: 1408 Permission denied - Failed to open old root directory
lxc_setup: 3419 Failed to pivot root into rootfs
do_start: 1218 Failed to setup container "102"
__sync_wait: 36 An error occurred in another process (expected sequence number 5)
__lxc_start: 1999 Failed to spawn container "102"
startup for container '102' failed

A new privileged container will start.

Sorry for maybe hijacking this topic, but it looks the same problem.

 

[Alex2801]

Thanks for your fast reply!

looking into the subvol it actually looks very empty...

#find /dataHDD0/encr/subvol-105-disk-0/
/dataHDD0/encr/subvol-105-disk-0/
/dataHDD0/encr/subvol-105-disk-0/mnt
/dataHDD0/encr/subvol-105-disk-0/mnt/backuptemp
/dataHDD0/encr/subvol-105-disk-0/mnt/ncdata
/dataHDD0/encr/subvol-105-disk-0/sys
/dataHDD0/encr/subvol-105-disk-0/dev
/dataHDD0/encr/subvol-105-disk-0/proc

Sadly i have currently no backup solution in place... I know, no backup no mercy

The good news: The whole data on the data subvol is still there, only the database is lost but thats kinda okay.

How can something like this happen? I had the container running till yesterday without any problems.

Edit: @BlackiE fyi it was a privileged container

Thanks
Alex

 

[fiona]

Hi,

Hi, suddenly also i have the same problem today. All my unprivileged containers won't start any more. I noticed that privileged containers will start. I updated and restarted my server as homelab. I tried to start a new unprivileged container;

# pct start 102
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 51(dev)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(full)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(null)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(random)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(tty)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(urandom)
__safe_mount_beneath_at: 1106 Function not implemented - Failed to open 54(zero)
lxc_pivot_root: 1408 Permission denied - Failed to open old root directory
lxc_setup: 3419 Failed to pivot root into rootfs
do_start: 1218 Failed to setup container "102"
__sync_wait: 36 An error occurred in another process (expected sequence number 5)
__lxc_start: 1999 Failed to spawn container "102"
startup for container '102' failed

A new privileged container will start.

Sorry for maybe hijacking this topic, but it looks the same problem.

maybe this thread will help.

Thanks for your fast reply!

looking into the subvol it actually looks very empty...

#find /dataHDD0/encr/subvol-105-disk-0/
/dataHDD0/encr/subvol-105-disk-0/
/dataHDD0/encr/subvol-105-disk-0/mnt
/dataHDD0/encr/subvol-105-disk-0/mnt/backuptemp
/dataHDD0/encr/subvol-105-disk-0/mnt/ncdata
/dataHDD0/encr/subvol-105-disk-0/sys
/dataHDD0/encr/subvol-105-disk-0/dev
/dataHDD0/encr/subvol-105-disk-0/proc

Sadly i have currently no backup solution in place... I know, no backup no mercy

The good news: The whole data on the data subvol is still there, only the database is lost but thats kinda okay.

How can something like this happen? I had the container running till yesterday without any problems.

Edit: @BlackiE fyi it was a privileged container

Thanks
Alex

Could you check your zpool history | grep 105? What does zfs list -o space dataHDD0/encr/subvol-105-disk-0 show?

 

[Alex2801]

Thanks for your answer!
I started up my machine a few minutes ago to check what you asked for and surprise: All my data in the subvolume is back and ofc the container is starting up again like its supposed to.

ls /dataHDD0/encr/subvol-105-disk-0
bin  boot  dev    etc  home  initrd.img  initrd.img.old  lib  lib64  media  mnt  opt  proc  root    run  sbin  srv    sys  tmp  usr  var  vmlinuz  vmlinuz.old

But to answer your questions, since im really interested what caused this i looked it up.

In the zpool history is only old stuff:

2019-09-29.18:35:15 zfs create -o acltype=posixacl -o xattr=sa -o refquota=31457280k dataHDD0/subvol-105-disk-0
2019-09-29.18:35:37 zfs destroy -r dataHDD0/subvol-105-disk-0
2019-09-29.18:37:02 zfs create -o acltype=posixacl -o xattr=sa -o refquota=31457280k dataHDD0/subvol-105-disk-0
2020-01-01.18:53:19 zfs destroy -r dataHDD0/subvol-105-disk-0
2020-01-01.21:42:13 zfs create -o acltype=posixacl -o xattr=sa -o refquota=1572864000k dataHDD0/encr/subvol-105-disk-0
2020-01-01.22:29:03 zfs destroy -r dataHDD0/encr/subvol-105-disk-0
2020-01-01.22:33:07 zfs create -o acltype=posixacl -o xattr=sa -o refquota=1572864000k dataHDD0/encr/subvol-105-disk-0
2020-01-01.23:18:03 zfs destroy -r dataHDD0/encr/subvol-105-disk-0
2020-01-01.23:20:18 zfs create -o acltype=posixacl -o xattr=sa -o refquota=31457280k dataHDD0/encr/subvol-105-disk-0
2020-01-01.23:21:22 zfs create -o acltype=posixacl -o xattr=sa -o refquota=1572864000k dataHDD0/encr/subvol-105-disk-1
2020-01-04.00:03:05 zfs set refquota=3145728000k dataHDD0/encr/subvol-105-disk-1
2020-06-14.19:08:25 zfs create -o acltype=posixacl -o xattr=sa -o refquota=104857600k dataHDD0/encr/subvol-105-disk-9
2020-06-22.19:50:19 zfs destroy -f dataHDD0/encr/subvol-105-disk-2
2020-06-22.19:50:25 zfs destroy -f dataHDD0/encr/subvol-105-disk-3
2020-06-22.19:50:30 zfs destroy -f dataHDD0/encr/subvol-105-disk-4
2020-06-22.19:50:41 zfs destroy -f dataHDD0/encr/subvol-105-disk-5
2020-06-22.19:50:47 zfs destroy -f dataHDD0/encr/subvol-105-disk-6
2020-06-22.19:50:52 zfs destroy -f dataHDD0/encr/subvol-105-disk-7
2020-06-22.19:51:14 zfs destroy -f dataHDD0/encr/subvol-105-disk-8
2019-12-11.21:37:05 zfs create -o acltype=posixacl -o xattr=sa -o refquota=31457280k dataSSD0/encr/subvol-105-disk-0
2019-12-19.23:43:59 zfs snapshot dataSSD0/encr/subvol-105-disk-0@prenewnextcloud
2020-01-01.22:29:03 zfs destroy -r dataSSD0/encr/subvol-105-disk-0
2020-01-01.22:32:37 zfs create -o acltype=posixacl -o xattr=sa -o refquota=31457280k dataSSD0/encr/subvol-105-disk-0
2020-01-01.23:18:03 zfs destroy -r dataSSD0/encr/subvol-105-disk-0

The listing shows:

# zfs list -o space dataHDD0/encr/subvol-105-disk-0
NAME                             AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
dataHDD0/encr/subvol-105-disk-0  9.63G  20.4G        0B   20.4G             0B         0B

But ofc i would be interested in what it was like yesterday when i encountered the error.

Is there any other log i should take a look on to determinate what this was?

Thank you all

Alex

 

[BlackiE]

Hi,

maybe this thread will help.



Could you check your zpool history | grep 105? What does zfs list -o space dataHDD0/encr/subvol-105-disk-0 show?

Many thanks. This thread helped and i solved the problem. I don't know what or me caused the problem. I can't remember if i maded some changes in the last few days. 4 days ago i restarted one container

 

[fiona]

Not sure what caused the problems either...
@Alex2801: it seems like the dataset was just not mounted (correctly). If you're lucky you'll find something in /var/log/syslog.
@BlackiE: Maybe check your shell's history.