LXC are not starting

krakazyabra · Jul 2, 2019

Hi!
Have such problem. After big transferring (db sync) and shutting down the LXC, I can't start it again.
The lxc-start log suddenly freezes on line

Code:

lxc-start 20126 20190702153043.716 DEBUG    network - network.c:instantiate_veth:206 - Instantiated veth "veth20126i0/vethTJF7WC", index is "53"
lxc-start 20126 20190702153043.716 TRACE    cgfsng - cgroups/cgfsng.c:cg_legacy_filter_and_set_cpus:434 - No isolated cpus detected
lxc-start 20126 20190702153043.716 DEBUG    cgfsng - cgroups/cgfsng.c:cg_legacy_handle_cpuset_hierarchy:619 - "cgroup.clone_children" was already set to "1"
lxc-start 20126 20190702153043.716 INFO     cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1537 - The container uses "lxc/20126" as cgroup

In another time, the lxc-start commands ends with ERROR:

Code:

lxc-start 20126 20190702114601.757 ERROR    network - network.c:instantiate_veth:106 - Operation not permitted - Failed to create veth pair "veth20126i0" and "vethC4D440"
lxc-start 20126 20190702114601.757 ERROR    network - network.c:lxc_create_network_priv:2462 - Failed to create network device
lxc-start 20126 20190702114601.758 ERROR    start - start.c:lxc_spawn:1646 - Failed to create the network
lxc-start 20126 20190702114601.758 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start 20126 20190702114601.758 TRACE    start - start.c:lxc_serve_state_clients:466 - Set container state to ABORTING
lxc-start 20126 20190702114601.758 TRACE    start - start.c:lxc_serve_state_clients:469 - No state clients registered
lxc-start 20126 20190702114601.758 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "20126"

After every unsuccessful start i do

Code:

root@proxmox2:/home/ep192# find /sys/fs/cgroup/*/lxc/20126 -depth -type d -print -delete
/sys/fs/cgroup/blkio/lxc/20126
/sys/fs/cgroup/cpuacct/lxc/20126
find: ‘/sys/fs/cgroup/cpu,cpuacct/lxc/20126’: No such file or directory
find: ‘/sys/fs/cgroup/cpu/lxc/20126’: No such file or directory
/sys/fs/cgroup/cpuset/lxc/20126
/sys/fs/cgroup/devices/lxc/20126
/sys/fs/cgroup/freezer/lxc/20126
/sys/fs/cgroup/hugetlb/lxc/20126
/sys/fs/cgroup/memory/lxc/20126
/sys/fs/cgroup/net_cls/lxc/20126
find: ‘/sys/fs/cgroup/net_cls,net_prio/lxc/20126’: No such file or directory
find: ‘/sys/fs/cgroup/net_prio/lxc/20126’: No such file or directory
/sys/fs/cgroup/perf_event/lxc/20126
/sys/fs/cgroup/pids/lxc/20126
/sys/fs/cgroup/rdma/lxc/20126
/sys/fs/cgroup/systemd/lxc/20126

So, the full log is:

Code:

root@proxmox2:/home/ep192# lxc-start -F -n 20126 --logfile lxc-start.log --logpriority TRACE
lxc-start: 20126: network.c: instantiate_veth: 106 Operation not permitted - Failed to create veth pair "veth20126i0" and "vethMSQ6ML"
                lxc-start: 20126: network.c: lxc_create_network_priv: 2462 Failed to create network device
                                                                                                          lxc-start: 20126: start.c: lxc_spawn: 1646 Failed to create the network
                                                           lxc-start: 20126: start.c: __lxc_start: 1989 Failed to spawn container "20126"
                   lxc-start: 20126: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: 20126: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

And logfile:

Code:

lxc-start 20126 20190702154526.238 TRACE    commands - commands.c:lxc_cmd:300 - Connection refused - Command "get_init_pid" failed to connect command socket
lxc-start 20126 20190702154526.238 TRACE    commands - commands.c:lxc_cmd:300 - Connection refused - Command "get_state" failed to connect command socket
lxc-start 20126 20190702154526.238 TRACE    commands - commands.c:lxc_cmd_init:1293 - Creating abstract unix socket "/var/lib/lxc/20126/command"
lxc-start 20126 20190702154526.238 TRACE    start - start.c:lxc_init_handler:760 - Unix domain socket 4 for command server is ready
lxc-start 20126 20190702154526.238 TRACE    start - start.c:lxc_start:2102 - Doing lxc_start
lxc-start 20126 20190702154526.238 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver AppArmor
lxc-start 20126 20190702154526.238 TRACE    start - start.c:lxc_init:779 - Initialized LSM
lxc-start 20126 20190702154526.238 TRACE    seccomp - seccomp.c:get_new_ctx:458 - Added arch 2 to main seccomp context
lxc-start 20126 20190702154526.238 TRACE    seccomp - seccomp.c:get_new_ctx:466 - Removed native arch from main seccomp context
lxc-start 20126 20190702154526.238 TRACE    seccomp - seccomp.c:get_new_ctx:458 - Added arch 3 to main seccomp context
lxc-start 20126 20190702154526.238 TRACE    seccomp - seccomp.c:get_new_ctx:466 - Removed native arch from main seccomp context
lxc-start 20126 20190702154526.238 TRACE    seccomp - seccomp.c:get_new_ctx:471 - Arch 4 already present in main seccomp context
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 1"
lxc-start 20126 20190702154526.238 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 1"
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 1"
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 1"
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 1"
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327681(errno)
lxc-start 20126 20190702154526.239 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
lxc-start 20126 20190702154526.239 TRACE    seccomp - seccomp.c:parse_config_v2:980 - Merged first compat seccomp context into main context
lxc-start 20126 20190702154526.239 TRACE    seccomp - seccomp.c:parse_config_v2:996 - Merged second compat seccomp context into main context
lxc-start 20126 20190702154526.239 TRACE    start - start.c:lxc_init:786 - Read seccomp policy
lxc-start 20126 20190702154526.239 TRACE    start - start.c:lxc_serve_state_clients:466 - Set container state to STARTING
lxc-start 20126 20190702154526.239 TRACE    start - start.c:lxc_serve_state_clients:469 - No state clients registered
lxc-start 20126 20190702154526.239 TRACE    start - start.c:lxc_init:794 - Set container state to "STARTING"
lxc-start 20126 20190702154526.239 TRACE    start - start.c:lxc_init:857 - Set environment variables
lxc-start 20126 20190702154526.239 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "20126", config section "lxc"
lxc-start 20126 20190702154526.734 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 20126 lxc pre-start with output: Plugin "PVE::Storage::Custom::LINSTORPlugin" is implementing an older storage API, an upgrade is recommended

lxc-start 20126 20190702154527.650 TRACE    start - start.c:lxc_init:864 - Ran pre-start hooks
lxc-start 20126 20190702154527.650 TRACE    start - start.c:setup_signal_fd:359 - Created signal file descriptor 5
lxc-start 20126 20190702154527.650 TRACE    start - start.c:lxc_init:875 - Set up signal fd
lxc-start 20126 20190702154527.651 DEBUG    terminal - terminal.c:lxc_terminal_peer_default:714 - Using terminal "/dev/tty" as proxy
lxc-start 20126 20190702154527.651 DEBUG    terminal - terminal.c:lxc_terminal_signal_init:192 - Created signal fd 9
lxc-start 20126 20190702154527.651 DEBUG    terminal - terminal.c:lxc_terminal_winsz:90 - Set window size to 118 columns and 60 rows
lxc-start 20126 20190702154527.651 TRACE    start - start.c:lxc_init:883 - Created console
lxc-start 20126 20190702154527.651 TRACE    start - start.c:lxc_init:890 - Chowned console
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1032 - basecginfo is:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1033 - 12:devices:/init.scope
11:memory:/init.scope
10:perf_event:/
9:cpu,cpuacct:/init.scope
8:hugetlb:/
7:pids:/init.scope
6:cpuset:/
5:rdma:/
4:blkio:/init.scope
3:net_cls,net_prio:/
2:freezer:/
1:name=systemd:/init.scope
0::/

lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 0: devices
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 1: memory
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 2: perf_event
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 3: cpu
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 4: cpuacct
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 5: hugetlb
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 6: pids
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 7: cpuset
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 8: rdma
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 9: blkio
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 10: net_cls
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 11: net_prio
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 12: freezer
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1036 - kernel subsystem 13: cgroup2
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_basecg_debuginfo:1039 - named subsystem 0: name=systemd
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:2701 - Writable cgroup hierarchies:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1013 -   Hierarchies:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   0: base_cgroup: /
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/systemd
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: name=systemd
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   1: base_cgroup: /
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/freezer
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: freezer
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   2: base_cgroup: /
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/net_cls,net_prio
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: net_cls
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       1: net_prio
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   3: base_cgroup: /
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/blkio
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: blkio
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   4: base_cgroup: /
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/rdma
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: rdma
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   5: base_cgroup: /
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/cpuset
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.652 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: cpuset
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   6: base_cgroup: /
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/pids
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: pids
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   7: base_cgroup: /
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/hugetlb
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: hugetlb
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   8: base_cgroup: /
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/cpu,cpuacct
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: cpu
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       1: cpuacct
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   9: base_cgroup: /
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/perf_event
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: perf_event
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   10: base_cgroup: /
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/memory
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: memory
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1018 -   11: base_cgroup: /
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1019 -       mountpoint:  /sys/fs/cgroup/devices
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1020 -       controllers:
lxc-start 20126 20190702154527.653 TRACE    cgfsng - cgroups/cgfsng.c:lxc_cgfsng_print_hierarchies:1022 -       0: devices
lxc-start 20126 20190702154527.653 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:56 - Initialized cgroup driver cgfsng
lxc-start 20126 20190702154527.653 TRACE    cgroup - cgroups/cgroup.c:cgroup_init:59 - Running with legacy cgroup layout
lxc-start 20126 20190702154527.653 TRACE    start - start.c:lxc_init:897 - Initialized cgroup driver
lxc-start 20126 20190702154527.657 TRACE    start - start.c:lxc_init:904 - Initialized LSM
lxc-start 20126 20190702154527.657 INFO     start - start.c:lxc_init:906 - Container "20126" is initialized
lxc-start 20126 20190702154527.658 ERROR    network - network.c:instantiate_veth:106 - Operation not permitted - Failed to create veth pair "veth20126i0" and "vethMSQ6ML"
lxc-start 20126 20190702154527.658 ERROR    network - network.c:lxc_create_network_priv:2462 - Failed to create network device
lxc-start 20126 20190702154527.658 ERROR    start - start.c:lxc_spawn:1646 - Failed to create the network
lxc-start 20126 20190702154527.658 DEBUG    network - network.c:lxc_delete_network:3185 - Deleted network devices
lxc-start 20126 20190702154527.658 TRACE    start - start.c:lxc_serve_state_clients:466 - Set container state to ABORTING
lxc-start 20126 20190702154527.658 TRACE    start - start.c:lxc_serve_state_clients:469 - No state clients registered
lxc-start 20126 20190702154527.658 ERROR    start - start.c:__lxc_start:1989 - Failed to spawn container "20126"
lxc-start 20126 20190702154527.658 TRACE    start - start.c:lxc_serve_state_clients:466 - Set container state to STOPPING
lxc-start 20126 20190702154527.658 TRACE    start - start.c:lxc_serve_state_clients:469 - No state clients registered
lxc-start 20126 20190702154527.783 TRACE    start - start.c:lxc_fini:1017 - Closed command socket
lxc-start 20126 20190702154527.783 TRACE    start - start.c:lxc_fini:1028 - Set container state to "STOPPED"
lxc-start 20126 20190702154527.783 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "20126", config section "lxc"
lxc-start 20126 20190702154528.286 INFO     conf - conf.c:run_script_argv:356 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "20126", config section "lxc"
lxc-start 20126 20190702154528.774 DEBUG    conf - conf.c:run_buffer:326 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 20126 lxc post-stop with output: Plugin "PVE::Storage::Custom::LINSTORPlugin" is implementing an older storage API, an upgrade is recommended

lxc-start 20126 20190702154529.930 ERROR    lxc_start - tools/lxc_start.c:main:330 - The container failed to start
lxc-start 20126 20190702154529.930 ERROR    lxc_start - tools/lxc_start.c:main:336 - Additional information can be obtained by setting the --logfile and --logpriority options

in the same time in dmesg:

Code:

[468828.487293] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: (null)
[468828.525300] audit: type=1400 audit(1562082327.652:5564): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxc-20126_</var/lib/lxc>" pid=28959 comm="apparmor_parser"
[468828.650983] audit: type=1400 audit(1562082327.776:5565): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxc-20126_</var/lib/lxc>" pid=28961 comm="apparmor_parser"
[468830.370294] vmbr0: port 10(veth20126i0) entered disabled state
[468830.370773] device veth20126i0 left promiscuous mode
[468830.370779] vmbr0: port 10(veth20126i0) entered disabled state

Unfortunatelly, I can't reboot the node, it is production with lots of LXC. CPU is fine, no IO Delay, another CTs are working well (afraid to touch them for tests).

kvaps · Jul 2, 2019

Strace hungs on

Code:

clone(

Code:

...
mkdir("/sys/fs/cgroup/", 0755)          = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/cpu,cpuacct//", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/cpu,cpuacct//lxc/", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/cpu,cpuacct//lxc/116", 0755) = 0
stat("/sys/fs/cgroup/perf_event//lxc/116", 0x7fff46531330) = -1 ENOENT (No such file or directory)
mkdir("/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/fs/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/", 0755)          = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/perf_event//", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/perf_event//lxc/", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/perf_event//lxc/116", 0755) = 0
stat("/sys/fs/cgroup/memory//lxc/116", 0x7fff46531330) = -1 ENOENT (No such file or directory)
mkdir("/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/fs/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/", 0755)          = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/memory//", 0755)  = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/memory//lxc/", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/memory//lxc/116", 0755) = 0
stat("/sys/fs/cgroup/devices//lxc/116", 0x7fff46531330) = -1 ENOENT (No such file or directory)
mkdir("/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/fs/", 0755)
= -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/", 0755)          = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/devices//", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/devices//lxc/", 0755) = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/devices//lxc/116", 0755) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/lib/lxc", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/lib/lxc/116", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/lib/lxc/116/rootfs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/lib/lxc/116/rootfs", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/lib/lxc/116/rootfs/.lxc-keep", O_RDWR|O_CREAT, 0600) = 14
fstatfs(14, {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=1016477, f_bfree=732825, f_bavail=676262, f_files=262416, f_ffree=234849, f_fsid={val=[2592929616, 3691259249]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
unlink("/var/lib/lxc/116/rootfs/.lxc-keep") = 0
clone(

oguz · Jul 3, 2019

hi,

you forgot `pveversion -v` and `pct config CTID`

krakazyabra · Jul 3, 2019

oguz said:
hi,

you forgot `pveversion -v` and `pct config CTID`

No problem:
`pveversion -v`

Code:

proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-7 (running version: 5.3-7/e8ed1e22)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.18-7-pve: 4.15.18-27
pve-kernel-4.15.18-1-pve: 4.15.18-19
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-19
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-35
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-33
pve-container: 2.0-33
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-6
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-44
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

`pct config CTID`

Code:

arch: amd64
cores: 8
hostname: zabbix-pxc-2
memory: 20480
net0: name=eth0,bridge=vmbr0,gw=10.28.0.1,hwaddr=66:46:E7:C4:EE:84,ip=10.28.20.126/16,ip6=dhcp,type=veth
ostype: centos
rootfs: proxmox2_local:20126/vm-20126-disk-0.raw,size=450G
swap: 2048

oguz · Jul 4, 2019

i can't seem to figure out the root cause here. the errors seem to be network related, so maybe it has something to do with ipv6. you can try disabling it to see if it solves anything. idk why exactly the network errors pop up in your case.

or maybe update everything and try again...

Search

Search

LXC are not starting

krakazyabra

Active Member

kvaps

Member

oguz

Proxmox Retired Staff

krakazyabra

Active Member

oguz

Proxmox Retired Staff