[SOLVED] All LXC containers have completely broken after host crashed

Linyu

Well-Known Member
Jun 14, 2019
43
2
48
25
Hi forum, I have a server running with 30 lxc containers, last night, the host crashed and rebooted without any signs, after that, the containers could not start at all.

Here are some information:

Code:
root@rvh-bt-gen2:~# pct config 13137
arch: amd64
cmode: shell
cores: 1
cpulimit: 1.0
hostname: RainYun-13137
memory: 512
nameserver: 114.114.114.114
net0: name=eth0,bridge=vmbr1,firewall=1,gw=10.0.0.3,hwaddr=0E:26:B3:6E:7E:A8,ip=10.0.0.91/24,type=veth
ostype: centos
rootfs: data:vm-13137-disk-0,size=5G
swap: 512
unprivileged: 1

Code:
root@rvh-bt-gen2:~# lxc-start -n 13137 -F
lxc-start: 13137: conf.c: run_buffer: 352 Script exited with status 1
lxc-start: 13137: conf.c: lxc_setup: 3656 Failed to run mount hooks
lxc-start: 13137: start.c: do_start: 1338 Failed to setup container "13137"
lxc-start: 13137: sync.c: __sync_wait: 62 An error occurred in another process (expected sequence number 5)
lxc-start: 13137: start.c: lxc_abort: 1133 Function not implemented - Failed to send SIGKILL to 8019
lxc-start: 13137: start.c: __lxc_start: 2080 Failed to spawn container "13137"
lxc-start: 13137: tools/lxc_start.c: main: 329 The container failed to start
lxc-start: 13137: tools/lxc_start.c: main: 335 Additional information can be obtained by setting the --logfile and --logpriority options

Code:
root@rvh-bt-gen2:~# pveversion -v
proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-5.3: 6.1-5
pve-kernel-helper: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
pve-kernel-5.3.13-3-pve: 5.3.13-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve4
libjs-extjs: 6.0.1-10
libknet1: 1.14-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-12
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-4
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-19
pve-docs: 6.1-6
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-10
pve-firmware: 3.0-5
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

Please help me because the system is not working at all now, it's a real problem.
 
Hi,

First of all, please upgrade your node to the current version.

Please start the LXC Container with this command: lxc-start -n 13137 -F -l DEBUG -o /tmp/lxc-13137.log and share the content of logs as attachment, and post output of systemctl status lxcfs.service please :)
 
Hi,

First of all, please upgrade your node to the current version.

Please start the LXC Container with this command: lxc-start -n 13137 -F -l DEBUG -o /tmp/lxc-13137.log and share the content of logs as attachment, and post output of systemctl status lxcfs.service please :)

Thank you very much.
After upgrade to the latest version, containers works fine now.
 
  • Like
Reactions: Moayad
Great! please mark the thread as [SOLVED] to help other people who have the same problem Thanks!

Have a nice day :)