The container failed to start

[kajdogaja]

Hello everybody,

I am new to proxmox and have an issue while trying to start a container.

after running following command:

lxc-start -n 120 -F -lDEBUG -o d.log

It says that container failed to start.
The content of d.log file is:

lxc-start 120 20190621082914.760 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 120 20190621082914.761 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 120 20190621082914.761 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.761 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 120 20190621082914.767 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 120 20190621082914.767 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 120 20190621082914.768 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "120", config section "lxc"
lxc-start 120 20190621082914.789 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start 120 20190621082914.789 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start 120 20190621082914.789 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 171 columns and 46 rows
lxc-start 120 20190621082914.913 INFO     start - start.c:lxc_init:906 - Container "120" is initialized
lxc-start 120 20190621082914.915 INFO     conf - conf.c:run_script:495 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "120", config section "net"
lxc-start 120 20190621082915.581 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/lxcnetaddbr 120 net up veth veth120i0 with output: Configuration file 'nodes/prox-02/lxc/120.conf' does not exist

lxc-start 120 20190621082915.590 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 2
lxc-start 120 20190621082915.641 ERROR    network - network.c:lxc_create_network_priv:2462 - Failed to create network device
lxc-start 120 20190621082915.641 ERROR    start - start.c:lxc_spawn:1646 - Failed to create the network
lxc-start 120 20190621082915.642 WARN     network - network.c:lxc_delete_network_priv:2594 - Operation not permitted - Failed to remove interface "eth0" with index 139
lxc-start 120 20190621082915.642 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start 120 20190621082915.642 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "120"
lxc-start 120 20190621082915.764 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "120", config section "lxc"
lxc-start 120 20190621082916.267 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "120", config section "lxc"
lxc-start 120 20190621082916.915 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 120 20190621082916.915 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

I would appreciate any help.

Thank You,
Luka

 

[t.lamprecht]

Your error is:

lxc-start 120 20190621082914.915 INFO conf - conf.c:run_script:495 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "120", config section "net" lxc-start 120 20190621082915.581 DEBUG conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/lxcnetaddbr 120 net up veth veth120i0 with output: Configuration file 'nodes/prox-02/lxc/120.conf' does not exist lxc-start 120 20190621082915.590 ERROR conf - conf.c:run_buffer:335 - Script exited with status 2

I.e., it boils down to:

Configuration file 'nodes/prox-02/lxc/120.conf' does not exist

So how you created the LXC containter? (WebUI, pct create, ...?) Did you moved file manually around?

 

[kajdogaja]

I believe it was created using web interface. The container actualy run fine for about a month. It is only yestery that the issue first appeared with vm as it was unresponsive and i could not ssh into it. My guess it would be that some process run at 100% cpu. After that i tried to stop it in order to restart it. Apperently even stop command failed.

I'm attaching image of tasks from web interface.

 

[kajdogaja]

After moving it to (using migrate option on web interface) to prox-02 i got different kind of error:

lxc-start 120 20190621093230.291 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 120 20190621093230.292 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 120 20190621093230.292 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 120 20190621093230.292 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 120 20190621093230.292 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "120", config section "lxc"
lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output: mount: wrong fs type, bad option, bad superblock on /dev/rbd8,

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:        missing codepage or helper program, or other error

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:        In some cases useful info is found in syslog - try

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:        dmesg | tail or so.

lxc-start 120 20190621093246.231 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output: command 'mount /dev/rbd8 /var/lib/lxc/120/rootfs//' failed: exit code 32

lxc-start 120 20190621093246.245 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 32
lxc-start 120 20190621093246.245 ERROR    start - start.c:lxc_init:861 - Failed to run lxc.hook.pre-start for container "120"
lxc-start 120 20190621093246.245 ERROR    start - start.c:__lxc_start:1944 - Failed to initialize container "120"
lxc-start 120 20190621093246.245 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 120 20190621093246.245 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

if i understead lof corretly there is a problem with file system? is there any way to fix that?

Thanks,
Luka

 

[t.lamprecht]

OK, so there a re a few more issues here on your system... If you have a support subscription I would heavily advise to go to our Enterprise Support and let them help you https://my.proxmox.com/index.php/en at least if this stuff is important and you're not to sure what the root causes of this are, also just a suggestions, naturally

Else I would look first if your ceph cluster this CT is backed on is healthy, as the log shows also write failures on a rados block device connected to this container..

If things "suddenly" break there should be some traces of the original root cause I'd look at the syslog / journalctl from around the time of suspected breakage.

 

[t.lamprecht]

if i understead lof corretly there is a problem with file system? is there any way to fix that?

depends, as said look at your ceph status, e.g. if this is a PVE hyperconverged ceph cluster you could also post

ceph -s

from any ceph node here, if it's a external cluster check that one.

 

[kajdogaja]

unfortunately my boss is out of office so i can't ask him if for support subscription credentials.

runnig

ceph -s

returns following:

2019-06-21 12:59:38.741856 7f8d6ad5a700 -1 Errors while parsing config file!
2019-06-21 12:59:38.741864 7f8d6ad5a700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-06-21 12:59:38.741865 7f8d6ad5a700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-06-21 12:59:38.741865 7f8d6ad5a700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)

I've also tried running dmesg | tail after trying to start and i get following entry:

[10153489.915571] EXT4-fs warning (device rbd8): ext4_multi_mount_protect:325: MMP interval 42 higher than expected, please wait.

[10153512.404186] EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: Device is already active on another node.
[10153512.404189] EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: MMP failure info: last update time: 1561114751, last update node: Prox-ve-02, last update device: rbd5

also journalctl -xe returns following error:

Jun 21 13:04:05 prox-02 mpt-statusd[47905]: detected non-optimal RAID status
Jun 21 13:04:08 prox-02 kernel: EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: Device is already active on another node.
Jun 21 13:04:08 prox-02 kernel: EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: MMP failure info: last update time: 1561115043, last update node: Prox-ve-02,
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory - Failed to receive the container state
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: tools/lxc_start.c: main: 330 The container failed to start
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority op
Jun 21 13:04:08 prox-02 pvestatd[2955]: unable to get PID for CT 120 (not running?)
Jun 21 13:04:08 prox-02 systemd[1]: pve-container@120.service: Control process exited, code=exited status=1
Jun 21 13:04:08 prox-02 pvedaemon[54381]: unable to get PID for CT 120 (not running?)
Jun 21 13:04:08 prox-02 systemd[1]: Failed to start PVE LXC Container: 120.
-- Subject: Unit pve-container@120.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pve-container@120.service has failed.
--
-- The result is failed.
Jun 21 13:04:08 prox-02 systemd[1]: pve-container@120.service: Unit entered failed state.
Jun 21 13:04:08 prox-02 systemd[1]: pve-container@120.service: Failed with result 'exit-code'.
Jun 21 13:04:08 prox-02 pvestatd[2955]: status update time (15.226 seconds)
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/common01: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/ssd: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/local-zfs: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/local: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/xenCeph: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/backup01: -1

Is it possible that this could occur doe to unsucessfuly stop?

Thanks,
Luka

 

[tom]

unfortunately my boss is out of office so i can't ask him if for support subscription credentials.

Submit your ticket on https://my.proxmox.com and register yourself with your company email address, we are able to find out your subscription status easily.