Single LXC container won't start after host reboot

[madsmao]

I have a single LXC container which is stubbornly refusing to start after a host reboot. All other LXC containers (15) and a single VM are running just fine. The host is updated, and it's identical to 3 other hosts in the cluster. There is quorum. No other hosts in the cluster are having this issue.

Here is what I get when trying to start the container in the foreground lxc-start -n 119 -F -l DEBUG -o /tmp/lxc-119.log:

close (rename) atomic file '/etc/sysconfig/network' failed: Operation not permitted
error in setup task PVE::LXC::Setup::pre_start_hook
lxc-start: conf.c: run_buffer: 405 Script exited with status 1.
lxc-start: start.c: lxc_init: 450 Failed to run lxc.hook.pre-start for container "119".
lxc-start: start.c: __lxc_start: 1321 Failed to initialize container "119".
lxc-start: tools/lxc_start.c: main: 366 The container failed to start.
lxc-start: tools/lxc_start.c: main: 370 Additional information can be obtained by setting the --logfile and --logpriority options.

And the output in /tmp/lxc-119.log looks like this:

lxc-start 20170529030101.685 INFO     lxc_start_ui - tools/lxc_start.c:main:275 - using rcfile /var/lib/lxc/119/config
lxc-start 20170529030101.686 WARN     lxc_confile - confile.c:config_pivotdir:1910 - lxc.pivotdir is ignored.  It will soon become an error.
lxc-start 20170529030101.686 WARN     lxc_start - start.c:lxc_check_inherited:238 - Inherited fd: 3.
lxc-start 20170529030101.686 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for reject_force_umount action 0.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for reject_force_umount action 0.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .[all].
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .kexec_load errno 1.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for kexec_load action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for kexec_load action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .open_by_handle_at errno 1.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for open_by_handle_at action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for open_by_handle_at action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .init_module errno 1.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for init_module action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for init_module action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .finit_module errno 1.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for finit_module action 327681.
lxc-start 20170529030101.686 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:270 - Seccomp: got negative for syscall: -10085: finit_module.
lxc-start 20170529030101.686 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:271 - This syscall will NOT be blacklisted.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for finit_module action 327681.
lxc-start 20170529030101.686 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:270 - Seccomp: got negative for syscall: -10085: finit_module.
lxc-start 20170529030101.686 WARN     lxc_seccomp - seccomp.c:do_resolve_add_rule:271 - This syscall will NOT be blacklisted.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .delete_module errno 1.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for delete_module action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for delete_module action 327681.
lxc-start 20170529030101.686 INFO     lxc_seccomp - seccomp.c:parse_config_v2:580 - Merging in the compat Seccomp ctx into the main one.
lxc-start 20170529030101.686 INFO     lxc_conf - conf.c:run_script_argv:424 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "119", config section "lxc".
lxc-start 20170529030101.957 ERROR    lxc_conf - conf.c:run_buffer:405 - Script exited with status 1.
lxc-start 20170529030101.957 ERROR    lxc_start - start.c:lxc_init:450 - Failed to run lxc.hook.pre-start for container "119".
lxc-start 20170529030101.957 ERROR    lxc_start - start.c:__lxc_start:1321 - Failed to initialize container "119".
lxc-start 20170529030101.957 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
lxc-start 20170529030101.957 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.

Seems like an issue with the file /etc/sysconfig/network inside the container (I assume it is, because it's a CentOS container), but I have no clue how to resolve it. Any help would be greatly appreciated!

 

[wbumiller]

Is it a privileged or unprivileged container?
Try to take a look at the file by mounting the container and inspecting it.

# pct mount 119
# ls -l /var/lib/lxc/119/rootfs/etc/sysconfig
# lsattr /var/lib/lxc/119/rootfs/etc/sysconfig/network

 

[madsmao]

Thanks, that helped!

Had to run chattr -i on /etc/sysconfig/network, /etc/hosts, and /etc/hostname, but then the container started just fine. Any idea what might have caused those files to get locked like that? Could it have simply been an unclean shutdown?