[SOLVED] Can't start LXC container

DavTDL

Member
May 31, 2019
10
0
6
32
Hello everybody,

I have rebooted my server to give it a hardware upgrade and when I booted it up, none of my two lxc containers will start.

pveversion -v:
Code:
proxmox-ve: 5.4-1 (running kernel: 4.15.18-15-pve)
pve-manager: 5.4-6 (running version: 5.4-6/aa7856c5)
pve-kernel-4.15: 5.4-3
pve-kernel-4.15.18-15-pve: 4.15.18-40
pve-kernel-4.15.18-12-pve: 4.15.18-36
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-10
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-52
libpve-guest-common-perl: 2.0-20
libpve-http-server-perl: 2.0-13
libpve-storage-perl: 5.0-43
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.1.0-3
lxcfs: 3.0.3-pve1
novnc-pve: 1.0.0-3
proxmox-widget-toolkit: 1.0-28
pve-cluster: 5.0-37
pve-container: 2.0-39
pve-docs: 5.4-2
pve-edk2-firmware: 1.20190312-1
pve-firewall: 3.0-22
pve-firmware: 2.0-6
pve-ha-manager: 2.0-9
pve-i18n: 1.1-4
pve-libspice-server1: 0.14.1-2
pve-qemu-kvm: 3.0.1-2
pve-xtermjs: 3.12.0-1
qemu-server: 5.0-52
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.13-pve1~bpo2

journalctl -xe

Code:
May 31 10:02:48 pve sshd[5146]: Received disconnect from 178.62.28.79 port 40464:11: Bye Bye [preauth]
May 31 10:02:48 pve sshd[5146]: Disconnected from 178.62.28.79 port 40464 [preauth]
May 31 10:03:00 pve systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
--
-- Unit pvesr.service has begun starting up.
May 31 10:03:01 pve systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.
May 31 10:03:08 pve sshd[5196]: Invalid user stephen from 134.209.11.199 port 60612
May 31 10:03:08 pve sshd[5196]: input_userauth_request: invalid user stephen [preauth]
May 31 10:03:08 pve sshd[5196]: pam_unix(sshd:auth): check pass; user unknown
May 31 10:03:08 pve sshd[5196]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=134.2
May 31 10:03:10 pve sshd[5196]: Failed password for invalid user stephen from 134.209.11.199 port 60612 ssh2
May 31 10:03:10 pve sshd[5196]: Received disconnect from 134.209.11.199 port 60612:11: Bye Bye [preauth]
May 31 10:03:10 pve sshd[5196]: Disconnected from 134.209.11.199 port 60612 [preauth]
May 31 10:04:00 pve systemd[1]: Starting Proxmox VE replication runner...
-- Subject: Unit pvesr.service has begun start-up
-- Defined-By: systemd
--
-- Unit pvesr.service has begun starting up.
May 31 10:04:01 pve systemd[1]: Started Proxmox VE replication runner.
-- Subject: Unit pvesr.service has finished start-up
-- Defined-By: systemd
--
-- Unit pvesr.service has finished starting up.
--
-- The start-up result is done.


systemctl status pve-container@102.service

Code:
● pve-container@102.service - PVE LXC Container: 102
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-05-31 09:56:59 EEST; 8min ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 4173 ExecStart=/usr/bin/lxc-start -n 102 (code=exited, status=1/FAILURE)

May 31 09:56:58 pve systemd[1]: Starting PVE LXC Container: 102...
May 31 09:56:59 pve lxc-start[4173]: lxc-start: 102: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory
May 31 09:56:59 pve lxc-start[4173]: lxc-start: 102: tools/lxc_start.c: main: 330 The container failed to start
May 31 09:56:59 pve lxc-start[4173]: lxc-start: 102: tools/lxc_start.c: main: 333 To get more details, run the container in f
May 31 09:56:59 pve lxc-start[4173]: lxc-start: 102: tools/lxc_start.c: main: 336 Additional information can be obtained by s
May 31 09:56:59 pve systemd[1]: pve-container@102.service: Control process exited, code=exited status=1
May 31 09:56:59 pve systemd[1]: Failed to start PVE LXC Container: 102.
May 31 09:56:59 pve systemd[1]: pve-container@102.service: Unit entered failed state.
May 31 09:56:59 pve systemd[1]: pve-container@102.service: Failed with result 'exit-code'.

My VM config file:
Code:
arch: amd64
cores: 1
hostname: fileserver
memory: 512
mp0: /data/storage,mp=/srv/storage
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=DE:36:5E:14:F8:42,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: LXC:subvol-100-disk-0,size=8G
swap: 512
unprivileged: 1

Hope this helps
 
Last edited:
Code:
lxc-start: 100: conf.c: run_buffer: 335 Script exited with status 2
lxc-start: 100: start.c: lxc_init: 861 Failed to run lxc.hook.pre-start for container "100"
lxc-start: 100: start.c: __lxc_start: 1944 Failed to initialize container "100"
lxc-start: 100: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: 100: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

Code:
lxc-start 100 20190531092822.661 INFO     confile - confile.c:set_config_idmaps:1673 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20190531092822.662 INFO     confile - confile.c:set_config_idmaps:1673 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20190531092822.662 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "keyctl errno 38"
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for keyctl action 327718(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for keyctl action 327718(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for keyctl action 327718(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for keyctl action 327718(errno)
lxc-start 100 20190531092822.663 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 100 20190531092822.663 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20190531092823.615 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start with output: unable to detect OS distribution

lxc-start 100 20190531092823.632 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 2
lxc-start 100 20190531092823.633 ERROR    start - start.c:lxc_init:861 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20190531092823.633 ERROR    start - start.c:__lxc_start:1944 - Failed to initialize container "100"
lxc-start 100 20190531092823.633 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 100 20190531092823.633 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options
 
Code:
lxc-start 100 20190531092823.615 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start with output: unable to detect OS distribution

this looks like the culprit.

according to your container configuration, your container is a debian. there should be a file in `/etc/debian_version` inside the container rootfs.

we can check this by mounting the container since it doesn't boot.

Code:
$ pct mount 100
mounted CT 100 in '/var/lib/lxc/100/rootfs'
$ cat /var/lib/lxc/100/rootfs/etc/debian_version
9.9

do you see the contents of the file when you follow the commands? should tell you the version.

the reason you get "unable to detect OS distribution" might be because this file was somehow deleted or your containers filesystem might be corrupted or similar. but idk why this would happen after a hardware upgrade unless you changed your disk... did you do something else?
 
The upgrades I have added is that I added Caddies for the disks because they were flying. I have put them in the exact order they were before.

I will check what you told me now
 
This was the output

Code:
root@pve:/var/lib/lxc/100/rootfs# cat /var/lib/lxc/100/rootfs/etc/debian_version
cat: /var/lib/lxc/100/rootfs/etc/debian_version: No such file or directory
 
is there any other file in the container at all??

it's weird that only this file would disappear/be deleted.

in any case, if that's the only missing file from the container, you can just try

Code:
echo "9.9" > /var/lib/lxc/100/rootfs/etc/debian_version

and then starting it... but how did this happen in the first place? are you sure the rootfs is intact?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!