[SOLVED] All LXC containers have completely broken after host crashed

[Linyu]

Hi forum, I have a server running with 30 lxc containers, last night, the host crashed and rebooted without any signs, after that, the containers could not start at all.

Here are some information:

root@rvh-bt-gen2:~# pct config 13137
arch: amd64
cmode: shell
cores: 1
cpulimit: 1.0
hostname: RainYun-13137
memory: 512
nameserver: 114.114.114.114
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.3,hwaddr=0E:26:B3:6E:7E:A8,ip=10.0.0.91/24,type=veth
ostype: centos
rootfs: data:vm-13137-disk-0,size=5G
swap: 512
unprivileged: 1

root@rvh-bt-gen2:~# lxc-start -n 13137 -F
lxc-start: 13137: conf.c: run_buffer: 352 Script exited with status 1
lxc-start: 13137: conf.c: lxc_setup: 3656 Failed to run mount hooks
lxc-start: 13137: start.c: do_start: 1338 Failed to setup container "13137"
lxc-start: 13137: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
lxc-start: 13137: start.c: lxc_abort: 1133 Function not implemented - Failed to send SIGKILL to 8019
lxc-start: 13137: start.c: __lxc_start: 2080 Failed to spawn container "13137"
lxc-start: 13137: tools/lxc_start.c: main: 329 The container failed to start
lxc-start: 13137: tools/lxc_start.c: main: 335 Additional information can be obtained by setting the --logfile and --logpriority options

root@rvh-bt-gen2:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-5
pve-kernel-helper: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.13-3-pve: 5.3.13-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve4
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-12
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-6
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-5
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Please help me because the system is not working at all now, it's a real problem.

 

[Moayad]

Hi,

First of all, please upgrade your node to the current version.

Please start the LXC Container with this command: lxc-start -n 13137 -F -l DEBUG -o /tmp/lxc-13137.log and share the content of logs as attachment, and post output of systemctl status lxcfs.service please

 

[Linyu]

Hi,

First of all, please upgrade your node to the current version.

Please start the LXC Container with this command: lxc-start -n 13137 -F -l DEBUG -o /tmp/lxc-13137.log and share the content of logs as attachment, and post output of systemctl status lxcfs.service please

Thank you very much.
After upgrade to the latest version, containers works fine now.

 

[Moayad]

Great! please mark the thread as [SOLVED] to help other people who have the same problem Thanks!

Have a nice day