[SOLVED] All but 1 container starts.

[taylor]

Have a few containers in Proxmox. All of them are standard Ubuntu 18.04 LXC. Recently had issues ssh-ing into one, (connection reset by peer) so I decided to stop it and start it again. It wouldn't start. Decided to upgrade Proxmox and reboot. Still couldn't get this one container to boot. I can mount the container w/o issue as similar threads request. Config file matches other lxc config files.

When I attempt to start the container from command line with debug and generate a log, this is the portion of the log containing the debug message and eventual error. There is a debug about /var/lib/lxc/102/rootfs/etc/lsb-release not existing, but that doesn't exist for any of the containers. Any help would be appreciated!


1827 for delete_module action 327681(errno)lxc-start 102 20181003035329.930 INFO seccomp - seccomp.carse_config_v2:953 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)lxc-start 102 20181003035329.930 INFO seccomp - seccomp.carse_config_v2:963 - Added native rule for arch -1073741762 for delete_module action 327681(errno)lxc-start 102 20181003035329.930 INFO seccomp - seccomp.carse_config_v2:967 - Merging compat seccomp contexts into main contextlxc-start 102 20181003035329.930 INFO conf - conf.c:run_script_argv:374 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "102", config section "lxc"lxc-start 102 20181003035330.318 DEBUG conf - conf.c:run_buffer:344 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 102 lxc pre-start with output: can't open '/var/lib/lxc/102/rootfs/etc/lsb-release' - No such file or directory
lxc-start 102 20181003035330.323 ERROR conf - conf.c:run_buffer:353 - Script exited with status 2
lxc-start 102 20181003035330.323 ERROR start - start.c:lxc_init:828 - Failed to run lxc.hook.pre-start for container "102"
lxc-start 102 20181003035330.324 ERROR start - start.c:__lxc_start:1914 - Failed to initialize container "102"
lxc-start 102 20181003035330.324 ERROR lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 102 20181003035330.324 ERROR lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

 

[dietmar]

What output do you get when starting lxc in foreground:

# lxc-start -n 102 -F

 

[taylor]

What output do you get when starting lxc in foreground:

# lxc-start -n 102 -F

Nothing. No output. Seems to process for a second and then goes to next line printing nothing out.

 

[wbumiller]

There is a debug about /var/lib/lxc/102/rootfs/etc/lsb-release not existing, but that doesn't exist for any of the containers.

Did you check using the above path or simply /etc/lsb-release from inside the containers? The above path is where the file systems get mounted to during startup of the container, and the file is part of ubuntu's `base-files` package, so it would be rather unusual for it not to exist.
Can you check the path above after doing a `pct mount 102`? (Afterwards don't forget to `pct unmount 102`
If you don't see any obvious issue there please post the complete debug log file as well as command output of `lxc-start -n 102 -F -lDEBUG -o lxc-start.log` and /etc/pve/lxc/102.conf

 

[taylor]

Yeah I'm checking the full path of /var/lib/lxc/102/rootfs/etc/lsb-release and there is nothing. I mounted and then checked and nothing is there either. Checking the other 2 containers I have running at the moment there is nothing at that location either and they currently have no issues. In fact, the two running containers don't have /etc/lsb-release under rootfs at all. Seems odd if it's required...

Here is the output of the command:

root@pver-linprox01:~# lxc-start -n 102 -F -lDEBUG -o lxc-start.log
lxc-start: 102: conf.c: run_buffer: 353 Script exited with status 2
lxc-start: 102: start.c: lxc_init: 828 Failed to run lxc.hook.pre-start for container "102"
lxc-start: 102: start.c: __lxc_start: 1914 Failed to initialize container "102"
lxc-start: 102: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: 102: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

Here is lxc-start.log (sorry new user, can't link to pastebin):

lxc-start 102 20181003130529.764 INFO     lsm - lsm/lsm.c:lsm_init:47 - LSM security driver AppArmor
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:do_resolve_add_rule:503 - Set seccomp rule to reject force umounts
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "[all]"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "kexec_load errno 1"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "open_by_handle_at errno 1"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "init_module errno 1"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "finit_module errno 1"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:757 - Processing "delete_module errno 1"
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:934 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:943 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:953 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:963 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 102 20181003130529.764 INFO     seccomp - seccomp.c:parse_config_v2:967 - Merging compat seccomp contexts into main context
lxc-start 102 20181003130529.764 INFO     conf - conf.c:run_script_argv:374 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "102", config section "lxc"
lxc-start 102 20181003130530.948 DEBUG    conf - conf.c:run_buffer:344 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 102 lxc pre-start with output: can't open '/var/lib/lxc/102/rootfs/etc/lsb-release' - No such file or directory

lxc-start 102 20181003130530.974 ERROR    conf - conf.c:run_buffer:353 - Script exited with status 2
lxc-start 102 20181003130530.974 ERROR    start - start.c:lxc_init:828 - Failed to run lxc.hook.pre-start for container "102"
lxc-start 102 20181003130530.974 ERROR    start - start.c:__lxc_start:1914 - Failed to initialize container "102"
lxc-start 102 20181003130530.974 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 102 20181003130530.974 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

Here is /etc/pve/lxc/102.conf:

arch: amd64
cores: 1
hostname: cver-linproxy01
memory: 512
net0: name=eth0,bridge=vmbr0,hwaddr=D6:1B:C1:24:7C:F4,ip=dhcp,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: local-lvm:vm-102-disk-1,size=4G
swap: 512

 

[wbumiller]

In fact, the two running containers don't have /etc/lsb-release under rootfs at all.

Dod you use `pct mount` as mentioned above before checking? Otherwise there won't be anything in there, so better check from inside the other containers and if you do find that file in them (and they are the same ubuntu version as the one that cannot start), see if you can copy it into the container by first using the `pct mount` command I mentioned above, making sure the expected data is in /var/lib/lxc/300/rootfs, and then copying the file into /var/...../etc/lsb-release. (You can also mount the running containers to make it easier to copy the file.)

 

[taylor]

Ah...good call. Makes sense now. I mounted a working container and can see everything properly.

Also checked the non-starting container again after mounting it and found the problem: lsb-release has been renamed lsb-releas (no "e"). Don't know how that happened, but I changed the name and all is well with the world.