Permission denied - Failed to exec "/sbin/init"

F

fresh

Member
Nov 28, 2022
17
0
6
Hello,

I am new to proxmox and LXCs.
One of my LXCs suddenly won't boot anymore. Three others are working fine.
I did some research on my own and found two possibilities: ZFS problems or a kernel update.
But I'm using ext4, so maybe it's the problem of an "apt full-upgrade"?

The error message is: ERROR start - ../src/lxc/start.c:start:2164 - Permission denied - Failed to exec "/sbin/init"

The full logs are here:
https://logpaste.com/O9QRerrR

Here is my pveversion:
proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

Can someone please help me?

Thank you in advance :-)
--
fresh
 
@Moayad

Here you are ;-)

root@pve:~# pct config 101
arch: amd64
cores: 4
features: nesting=1
hostname: pveDocker
lock: mounted
memory: 8192
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.XX.1,hwaddr=XX:XX:XX:XX:XX,ip=192.168.XX.101/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-101-disk-0,size=256G
swap: 8192
unprivileged: 1
 
Hello,

Code: 
ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)

I would check if the "init" exists in "/var/lib/lxc/101/rootfs/sbin/init" by doing the following in your PVE host:

Bash: 
pct mount 101
ls -la /var/lib/lxc/101/rootfs/sbin/init
 
@Moayad here you are ;-)
Code: 
root@pve:~# ls -la /var/lib/lxc/101/rootfs/sbin/init
lrwxrwxrwx 1 100100 100101 20 Sep  9 20:47 /var/lib/lxc/101/rootfs/sbin/init -> /lib/systemd/systemd

By the way: I've already done it the hard way. Data backup via mount 101. LXC destroy. LXC created again -> same problem after a few hours...
It starts with a permission error and after restarting the PVE host I can't start the LXC anymore.
 
Last edited:
What template you are using on this container? Can you reproduce the issue on a new CT? If yes, please post the steps.
 
@Moayad
I use the default ubuntu template: ubuntu-22.04-standard_22.04-1_amd64.tar.zst

I can reproduce on a new CT. But I can't trace it down completely. Because this error appears after rebooting the host. So it's very difficult to chase the exact problem.
 
I have the same exact problem with a container I created using a pveam-downloaded template:

Code: 
root@proxmox:~# pveam list local
NAME                                                         SIZE
local:vztmpl/alpine-3.21-default_20241217_amd64.tar.xz       3.01MB

The template itself obviously does contain the sbin/init:
Code: 
root@proxmox:/var/lib/vz/template/cache# tar xvf alpine-3.21-default_20241217_amd64.tar.xz |grep sbin\/init
./sbin/init

And the container I recovered does contain sbin/init as well:

Code: 
root@proxmox:/pool1/subvol-122-disk-0/sbin# ls -al|grep init
lrwxrwxrwx  1 100000 100000    12 Dec  5 13:17 init -> /bin/busybox

However, the container fails to start with:
Code: 
ERROR    start - ../src/lxc/start.c:start:2204 - Permission denied - Failed to exec "/sbin/init"
ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)
INFO     network - ../src/lxc/network.c:lxc_delete_network_priv:3720 - Removed interface "veth122i0" from ""
DEBUG    network - ../src/lxc/network.c:lxc_delete_network:4217 - Deleted network devices
ERROR    start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "122"
WARN     start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 16 for process 483011

The sbin/init symbolic link is present:
Code: 
root@proxmox:/pool1/subvol-122-disk-0/sbin# ls -al|grep init
lrwxrwxrwx  1 100000 100000    12 Dec  5 13:17 init -> /bin/busybox

And so is the busybox target:
Code: 
root@proxmox:/pool1/subvol-122-disk-0/bin# ls -al busybox
-rwxr-xr-x 1 100000 100000 808712 Jan 17 19:12 busybox

Except:
Code: 
root@proxmox:/pool1/subvol-122-disk-0/bin# ./busybox
-bash: ./busybox: Permission denied

One notable thing is that I restored to another ZFS pool than the original container was stored at. So I checked the zfs exec flag and, not surprisingly, it is off:
Code: 
zfs get exec pool1/subvol-122-disk-0
NAME                     PROPERTY  VALUE  SOURCE
pool1/subvol-122-disk-0  exec      off    inherited from pool1

As is the flag on this whole pool, as indicated.

So there issue here is that when adding a ZFS storage and enabling Container content, Proxmox should make sure the exec property gets switched on.

I filed a bug upstream: https://bugzilla.proxmox.com/show_bug.cgi?id=6281
 
Last edited:
You must log in or register to reply here.
Top Bottom