Permission denied - Failed to exec "/sbin/init"

[fresh]

Hello,

I am new to proxmox and LXCs.
One of my LXCs suddenly won't boot anymore. Three others are working fine.
I did some research on my own and found two possibilities: ZFS problems or a kernel update.
But I'm using ext4, so maybe it's the problem of an "apt full-upgrade"?

The error message is: ERROR start - ../src/lxc/start.c:start:2164 - Permission denied - Failed to exec "/sbin/init"

The full logs are here:
https://logpaste.com/O9QRerrR

Here is my pveversion:

Spoiler

proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.60-2-pve: 5.15.60-2
pve-kernel-5.15.30-2-pve: 5.15.30-3
ceph-fuse: 15.2.16-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

Can someone please help me?

Thank you in advance
--
fresh

 

[Moayad]

Hello,

Can you please post the LXC config (pct config <CTID>) as well?

 

[fresh]

@Moayad

Here you are ;-)

Spoiler

root@pve:~# pct config 101
arch: amd64
cores: 4
features: nesting=1
hostname: pveDocker
lock: mounted
memory: 8192
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.XX.1,hwaddr=XX:XX:XX:XX:XX,ip=192.168.XX.101/24,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-101-disk-0,size=256G
swap: 8192
unprivileged: 1

 

[Moayad]

Hello,

ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)

I would check if the "init" exists in "/var/lib/lxc/101/rootfs/sbin/init" by doing the following in your PVE host:

pct mount 101
ls -la /var/lib/lxc/101/rootfs/sbin/init

 

[fresh]

@Moayad here you are ;-)

root@pve:~# ls -la /var/lib/lxc/101/rootfs/sbin/init
lrwxrwxrwx 1 100100 100101 20 Sep  9 20:47 /var/lib/lxc/101/rootfs/sbin/init -> /lib/systemd/systemd

By the way: I've already done it the hard way. Data backup via mount 101. LXC destroy. LXC created again -> same problem after a few hours...
It starts with a permission error and after restarting the PVE host I can't start the LXC anymore.

 

[fresh]

Any ideas?

 

[Moayad]

What template you are using on this container? Can you reproduce the issue on a new CT? If yes, please post the steps.

 

[fresh]

@Moayad
I use the default ubuntu template: ubuntu-22.04-standard_22.04-1_amd64.tar.zst

I can reproduce on a new CT. But I can't trace it down completely. Because this error appears after rebooting the host. So it's very difficult to chase the exact problem.

 

[fresh]

Nobody? It's annoying, because it reoccurs on any LXC clone after a few hours.