The container failed to start

kajdogaja

New Member
Jun 21, 2019
4
0
1
114
Hello everybody,

I am new to proxmox and have an issue while trying to start a container.

after running following command:
Code:
lxc-start -n 120 -F -lDEBUG -o d.log

It says that container failed to start.
The content of d.log file is:
Code:
lxc-start 120 20190621082914.760 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 120 20190621082914.761 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 120 20190621082914.761 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.761 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621082914.762 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 120 20190621082914.763 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 120 20190621082914.764 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 120 20190621082914.765 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 120 20190621082914.766 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 120 20190621082914.767 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 120 20190621082914.767 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 120 20190621082914.768 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "120", config section "lxc"
lxc-start 120 20190621082914.789 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start 120 20190621082914.789 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start 120 20190621082914.789 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 171 columns and 46 rows
lxc-start 120 20190621082914.913 INFO     start - start.c:lxc_init:906 - Container "120" is initialized
lxc-start 120 20190621082914.915 INFO     conf - conf.c:run_script:495 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "120", config section "net"
lxc-start 120 20190621082915.581 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/lxcnetaddbr 120 net up veth veth120i0 with output: Configuration file 'nodes/prox-02/lxc/120.conf' does not exist

lxc-start 120 20190621082915.590 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 2
lxc-start 120 20190621082915.641 ERROR    network - network.c:lxc_create_network_priv:2462 - Failed to create network device
lxc-start 120 20190621082915.641 ERROR    start - start.c:lxc_spawn:1646 - Failed to create the network
lxc-start 120 20190621082915.642 WARN     network - network.c:lxc_delete_network_priv:2594 - Operation not permitted - Failed to remove interface "eth0" with index 139
lxc-start 120 20190621082915.642 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start 120 20190621082915.642 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "120"
lxc-start 120 20190621082915.764 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "120", config section "lxc"
lxc-start 120 20190621082916.267 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "120", config section "lxc"
lxc-start 120 20190621082916.915 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 120 20190621082916.915 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

I would appreciate any help.

Thank You,
Luka
 
Your error is:
lxc-start 120 20190621082914.915 INFO conf - conf.c:run_script:495 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "120", config section "net" lxc-start 120 20190621082915.581 DEBUG conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/lxcnetaddbr 120 net up veth veth120i0 with output: Configuration file 'nodes/prox-02/lxc/120.conf' does not exist lxc-start 120 20190621082915.590 ERROR conf - conf.c:run_buffer:335 - Script exited with status 2

I.e., it boils down to:
Configuration file 'nodes/prox-02/lxc/120.conf' does not exist

So how you created the LXC containter? (WebUI, pct create, ...?) Did you moved file manually around?
 
I believe it was created using web interface. The container actualy run fine for about a month. It is only yestery that the issue first appeared with vm as it was unresponsive and i could not ssh into it. My guess it would be that some process run at 100% cpu. After that i tried to stop it in order to restart it. Apperently even stop command failed.

I'm attaching image of tasks from web interface.
 

Attachments

  • Capture.PNG
    Capture.PNG
    21 KB · Views: 13
After moving it to (using migrate option on web interface) to prox-02 i got different kind of error:
Code:
lxc-start 120 20190621093230.291 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 120 20190621093230.291 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 120 20190621093230.292 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 120 20190621093230.292 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 120 20190621093230.292 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 120 20190621093230.292 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "120", config section "lxc"
lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output: mount: wrong fs type, bad option, bad superblock on /dev/rbd8,

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:        missing codepage or helper program, or other error

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:        In some cases useful info is found in syslog - try

lxc-start 120 20190621093246.230 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output:        dmesg | tail or so.

lxc-start 120 20190621093246.231 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 120 lxc pre-start with output: command 'mount /dev/rbd8 /var/lib/lxc/120/rootfs//' failed: exit code 32

lxc-start 120 20190621093246.245 ERROR    conf - conf.c:run_buffer:335 - Script exited with status 32
lxc-start 120 20190621093246.245 ERROR    start - start.c:lxc_init:861 - Failed to run lxc.hook.pre-start for container "120"
lxc-start 120 20190621093246.245 ERROR    start - start.c:__lxc_start:1944 - Failed to initialize container "120"
lxc-start 120 20190621093246.245 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 120 20190621093246.245 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

if i understead lof corretly there is a problem with file system? is there any way to fix that?

Thanks,
Luka
 
OK, so there a re a few more issues here on your system... If you have a support subscription I would heavily advise to go to our Enterprise Support and let them help you https://my.proxmox.com/index.php/en at least if this stuff is important and you're not to sure what the root causes of this are, also just a suggestions, naturally :)

Else I would look first if your ceph cluster this CT is backed on is healthy, as the log shows also write failures on a rados block device connected to this container..

If things "suddenly" break there should be some traces of the original root cause I'd look at the syslog / journalctl from around the time of suspected breakage.
 
if i understead lof corretly there is a problem with file system? is there any way to fix that?
depends, as said look at your ceph status, e.g. if this is a PVE hyperconverged ceph cluster you could also post
Code:
ceph -s
from any ceph node here, if it's a external cluster check that one.
 
unfortunately my boss is out of office so i can't ask him if for support subscription credentials.

runnig
Code:
ceph -s
returns following:
Code:
2019-06-21 12:59:38.741856 7f8d6ad5a700 -1 Errors while parsing config file!
2019-06-21 12:59:38.741864 7f8d6ad5a700 -1 parse_file: cannot open /etc/ceph/ceph.conf: (2) No such file or directory
2019-06-21 12:59:38.741865 7f8d6ad5a700 -1 parse_file: cannot open ~/.ceph/ceph.conf: (2) No such file or directory
2019-06-21 12:59:38.741865 7f8d6ad5a700 -1 parse_file: cannot open ceph.conf: (2) No such file or directory
Error initializing cluster client: ObjectNotFound('error calling conf_read_file',)

I've also tried running dmesg | tail after trying to start and i get following entry:
Code:
[10153489.915571] EXT4-fs warning (device rbd8): ext4_multi_mount_protect:325: MMP interval 42 higher than expected, please wait.

[10153512.404186] EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: Device is already active on another node.
[10153512.404189] EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: MMP failure info: last update time: 1561114751, last update node: Prox-ve-02, last update device: rbd5

also journalctl -xe returns following error:
Code:
Jun 21 13:04:05 prox-02 mpt-statusd[47905]: detected non-optimal RAID status
Jun 21 13:04:08 prox-02 kernel: EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: Device is already active on another node.
Jun 21 13:04:08 prox-02 kernel: EXT4-fs warning (device rbd8): ext4_multi_mount_protect:338: MMP failure info: last update time: 1561115043, last update node: Prox-ve-02,
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: lxccontainer.c: wait_on_daemonized_start: 856 No such file or directory - Failed to receive the container state
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: tools/lxc_start.c: main: 330 The container failed to start
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
Jun 21 13:04:08 prox-02 lxc-start[46825]: lxc-start: 120: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority op
Jun 21 13:04:08 prox-02 pvestatd[2955]: unable to get PID for CT 120 (not running?)
Jun 21 13:04:08 prox-02 systemd[1]: pve-container@120.service: Control process exited, code=exited status=1
Jun 21 13:04:08 prox-02 pvedaemon[54381]: unable to get PID for CT 120 (not running?)
Jun 21 13:04:08 prox-02 systemd[1]: Failed to start PVE LXC Container: 120.
-- Subject: Unit pve-container@120.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit pve-container@120.service has failed.
--
-- The result is failed.
Jun 21 13:04:08 prox-02 systemd[1]: pve-container@120.service: Unit entered failed state.
Jun 21 13:04:08 prox-02 systemd[1]: pve-container@120.service: Failed with result 'exit-code'.
Jun 21 13:04:08 prox-02 pvestatd[2955]: status update time (15.226 seconds)
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/common01: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/ssd: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/local-zfs: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/local: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/xenCeph: -1
Jun 21 13:04:08 prox-02 pmxcfs[44774]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-storage/prox-02/backup01: -1

Is it possible that this could occur doe to unsucessfuly stop?

Thanks,
Luka
 
unfortunately my boss is out of office so i can't ask him if for support subscription credentials.

Submit your ticket on https://my.proxmox.com and register yourself with your company email address, we are able to find out your subscription status easily.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!