LXC won't start/stop - Failed to create veth pair

breakingspell

Member
Nov 21, 2017
2
0
6
30
Have an odd issue on a non-production server i've been running for a few months. Starting/restarting a container (ID 101 for example) while any other container is active will cause the 2nd CT's network pair to be generated as veth101i0@vethXXYYZZ rather than veth101i0@ifX as it usually does, and the container will not start.

It can happen both using the gui or pct to manage the containers, and any containers will exhibit the behavior, even after stopping all of them and clearing the hung lxc processes and veth pair manually. It can occur with any combination of containers, as long as is one is active, no other containers can be started.

Furthermore, other lxc processes will hang, causing the web UI to lose data (screenshot) and operations will timeout on other containers until the offending lxc-start and lxc-monitor processes for ID 101 are killed. This behavior isn't affected by running qemu machines, and those can still be controlled via ssh and other means while the proxmox gui is unresponsive.

The issue occurred from the start as I set the server up, but was resolved when I downgraded my kernel shortly afterwards to pve-kernel-4.10.17-5-pve as detailed for the quick fix in this seemingly-unrelated issue before the fix was posted in the upstream kernel.
I had forgotten to reconfigure grub to use the latest kernel rather than pve-kernel-4.10.17-5-pve until this weekend, and the container issue has reoccurred with pve-kernel-4.13.13-4-pve. I can boot between the two kernels and the issue indeed vanishes on 4.10.17-5-pve, it'll start any combination of containers and VMs under that kernel.

My configs are as follows, let me know if there's any other logs I can gather to help better explain the issue. It's described to a T in two github reports that I cannot link to due to this being my first post, but they're from years ago and never received resolutions. Any help is appreciated and thanks in advance!

Code:
pveversion -v

proxmox-ve: 5.1-35 (running kernel: 4.13.13-4-pve)
pve-manager: 5.1-42 (running version: 5.1-42/724a6cb3)
pve-kernel-4.13.13-4-pve: 4.13.13-35
pve-kernel-4.13.13-2-pve: 4.13.13-33
pve-kernel-4.13.8-3-pve: 4.13.8-30
pve-kernel-4.13.13-3-pve: 4.13.13-34
pve-kernel-4.10.17-5-pve: 4.10.17-25
pve-kernel-4.13.13-1-pve: 4.13.13-31
libpve-http-server-perl: 2.0-8
lvm2: 2.02.168-pve6
corosync: 2.4.2-pve3
libqb0: 1.0.1-1
pve-cluster: 5.0-19
qemu-server: 5.0-19
pve-firmware: 2.0-3
libpve-common-perl: 5.0-25
libpve-guest-common-perl: 2.0-14
libpve-access-control: 5.0-7
libpve-storage-perl: 5.0-17
pve-libspice-server1: 0.12.8-3
vncterm: 1.5-3
pve-docs: 5.1-16
pve-qemu-kvm: 2.9.1-5
pve-container: 2.0-18
pve-firewall: 3.0-5
pve-ha-manager: 2.0-4
ksm-control-daemon: 1.2-2
glusterfs-client: 3.8.8-1
lxc-pve: 2.1.1-2
lxcfs: 2.0.8-1
criu: 2.11.1-1~bpo90
novnc-pve: 0.6-4
smartmontools: 6.5+svn4324-1
zfsutils-linux: 0.7.3-pve1~bpo9

Code:
/etc/pve/lxc/101.conf

arch: amd64
cores: 1
hostname: BackboneA1
memory: 512
net0: name=eth0,bridge=vmbr0,gw=192.168.1.1,hwaddr=12:CE:3C:63:64:56,ip=192.168.1.101/24,ip6=auto,type=veth
ostype: debian
rootfs: vmstorage:subvol-101-disk-4,size=8G
swap: 512

Code:
/etc/network/interfaces on host

auto lo
iface lo inet loopback
iface eno1 inet manual
auto vmbr0
iface vmbr0 inet static
        address 192.168.1.66
        netmask 255.255.255.0
        gateway 192.168.1.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0
iface enp2s0 inet manual


/etc/network/interfaces on container

auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
        address 192.168.1.101
        netmask 255.255.255.0
        gateway 192.168.1.1
iface eth0 inet6 auto

Code:
ip link

9: veth101i0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:68:82:74:3f:3e brd ff:ff:ff:ff:ff:ff link-netnsid 1

with 2 containers

13: veth100i0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether fe:fd:6b:0e:5a:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
16: vethKM87M1@veth101i0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 92:df:a4:85:5c:f0 brd ff:ff:ff:ff:ff:ff
17: veth101i0@vethKM87M1: <NO-CARRIER,BROADCAST,MULTICAST,UP,M-DOWN> mtu 1500 qdisc noqueue master vmbr0 state LOWERLAYERDOWN mode DEFAULT group default qlen 1000
    link/ether fe:69:ad:11:1a:3d brd ff:ff:ff:ff:ff:ff

Code:
lxc-start -F -n 101 --logfile=101.log --logpriority=debug

      lxc-start 101 20180115032616.916 INFO     lxc_start_ui - tools/lxc_start.c:main:280 - using rcfile /var/lib/lxc/101/config
      lxc-start 101 20180115032616.916 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for reject_force_umount action 0(kill).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for reject_force_umount action 0(kill).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .[all].
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .kexec_load errno 1.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for kexec_load action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for kexec_load action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .open_by_handle_at errno 1.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for open_by_handle_at action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for open_by_handle_at action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .init_module errno 1.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for init_module action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for init_module action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .finit_module errno 1.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for finit_module action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for finit_module action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .delete_module errno 1.
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for delete_module action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for delete_module action 327681(errno).
      lxc-start 101 20180115032616.916 INFO     lxc_seccomp - seccomp.c:parse_config_v2:624 - Merging in the compat Seccomp ctx into the main one.
      lxc-start 101 20180115032616.917 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "101", config section "lxc".
      lxc-start 101 20180115032617.216 DEBUG    lxc_start - start.c:setup_signal_fd:301 - Set SIGCHLD handler with file descriptor: 5.
      lxc-start 101 20180115032617.217 DEBUG    console - console.c:lxc_console_peer_default:459 - using "/dev/tty" as peer tty device
      lxc-start 101 20180115032617.217 DEBUG    console - console.c:lxc_console_sigwinch_init:151 - process 29307 created signal fd 9 to handle SIGWINCH events
      lxc-start 101 20180115032617.217 DEBUG    console - console.c:lxc_console_winsz:71 - set winsz dstfd:6 cols:125 rows:41
      lxc-start 101 20180115032617.217 INFO     lxc_start - start.c:lxc_init:680 - container "101" is initialized
      lxc-start 101 20180115032617.217 INFO     lxc_conf - conf.c:run_script:507 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "101", config section "net".
      lxc-start 101 20180115032617.533 DEBUG    lxc_network - network.c:instantiate_veth:219 - Instantiated veth "veth101i0/vethQKBDE3", index is "14"
      lxc-start 101 20180115032617.533 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:67 - cgroup driver cgroupfs-ng initing for 101
      lxc-start 101 20180115032617.534 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:filter_and_set_cpus:469 - No isolated cpus detected.
      lxc-start 101 20180115032617.534 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:handle_cpuset_hierarchy:640 - "cgroup.clone_children" was already set to "1".
      lxc-start 101 20180115032617.534 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWNS.
      lxc-start 101 20180115032617.534 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWPID.
      lxc-start 101 20180115032617.534 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWUTS.
      lxc-start 101 20180115032617.534 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWIPC.
      lxc-start 101 20180115032617.534 INFO     lxc_start - start.c:lxc_spawn:1259 - Cloned CLONE_NEWNET.
      lxc-start 101 20180115032617.534 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'memory.limit_in_bytes' set to '536870912'
      lxc-start 101 20180115032617.534 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'memory.memsw.limit_in_bytes' set to '1073741824'
      lxc-start 101 20180115032617.534 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'cpu.shares' set to '1024'
      lxc-start 101 20180115032617.535 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'cpuset.cpus' set to '4'
      lxc-start 101 20180115032617.535 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2137 - cgroup has been setup
      lxc-start 101 20180115032617.557 DEBUG    lxc_network - network.c:lxc_network_move_created_netdev_priv:2450 - Moved network device "vethQKBDE3"/"eth0" to network namespace of 29477
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.deny' set to 'a'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c *:* m'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'b *:* m'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 1:3 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 1:5 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 1:7 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 5:0 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 5:1 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 5:2 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 1:8 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 1:9 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 136:* rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 10:229 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 254:0 rm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 10:200 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 10:228 rwm'
      lxc-start 101 20180115032617.557 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2132 - cgroup 'devices.allow' set to 'c 10:232 rwm'
      lxc-start 101 20180115032617.557 INFO     lxc_cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2137 - cgroup has been setup
      lxc-start 101 20180115032617.558 INFO     lxc_start - start.c:do_start:925 - Unshared CLONE_NEWCGROUP.
      lxc-start 101 20180115032617.558 DEBUG    storage - storage/storage.c:storage_query:252 - Detected rootfs type "dir"
      lxc-start 101 20180115032617.558 DEBUG    lxc_conf - conf.c:lxc_setup_rootfs:1319 - Mounted rootfs "/var/lib/lxc/101/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)".
      lxc-start 101 20180115032617.558 INFO     lxc_conf - conf.c:setup_utsname:768 - 'BackboneA1' hostname has been setup
      lxc-start 101 20180115032617.573 DEBUG    lxc_network - network.c:setup_hw_addr:2716 - Mac address "12:CE:3C:63:64:56" on "eth0" has been setup
      lxc-start 101 20180115032617.573 DEBUG    lxc_network - network.c:lxc_setup_netdev_in_child_namespaces:2974 - Network device "eth0" has been setup
      lxc-start 101 20180115032617.573 INFO     lxc_network - network.c:lxc_setup_network_in_child_namespaces:3002 - network has been setup
      lxc-start 101 20180115032617.573 INFO     lxc_conf - conf.c:mount_autodev:1149 - Preparing "/dev"
      lxc-start 101 20180115032617.573 INFO     lxc_conf - conf.c:mount_autodev:1171 - Mounted tmpfs on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev"
      lxc-start 101 20180115032617.573 INFO     lxc_conf - conf.c:mount_autodev:1188 - Prepared "/dev"
      lxc-start 101 20180115032617.573 DEBUG    lxc_conf - conf.c:mount_entry:1863 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
      lxc-start 101 20180115032617.573 DEBUG    lxc_conf - conf.c:mount_entry:1884 - Flags for "/sys/fs/fuse/connections" were 4096, required extra flags are 0
      lxc-start 101 20180115032617.573 DEBUG    lxc_conf - conf.c:mount_entry:1894 - Mountflags already were 4096, skipping remount
      lxc-start 101 20180115032617.573 DEBUG    lxc_conf - conf.c:mount_entry:1921 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
      lxc-start 101 20180115032617.573 INFO     lxc_conf - conf.c:mount_file_entries:2212 - Set up mount entries
      lxc-start 101 20180115032617.573 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "101", config section "lxc".
      lxc-start 101 20180115032617.583 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxc/hooks/lxc-pve-autodev-hook" for container "101", config section "lxc".
      lxc-start 101 20180115032617.695 INFO     lxc_conf - conf.c:lxc_fill_autodev:1224 - Populating "/dev"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/null"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/zero"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/full"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/urandom"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/random"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_fill_autodev:1270 - Created device node "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/tty"
      lxc-start 101 20180115032617.695 INFO     lxc_conf - conf.c:lxc_fill_autodev:1275 - Populated "/dev"
      lxc-start 101 20180115032617.695 DEBUG    lxc_conf - conf.c:lxc_setup_dev_console:1603 - mounted pts device "/dev/pts/5" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/console"
      lxc-start 101 20180115032617.695 INFO     lxc_utils - utils.c:lxc_mount_proc_if_needed:1758 - I am 1, /proc/self points to "1"
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_rootfs_pivot_root:1127 - pivot_root syscall to '/usr/lib/x86_64-linux-gnu/lxc/rootfs' successful
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_pivot_root:1436 - finished pivot root
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_setup_devpts:1481 - mount new devpts instance with options "newinstance,ptmxmode=0666,mode=0620,gid=5,max=1024"
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_setup_devpts:1501 - created dummy "/dev/ptmx" file as bind mount target
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_setup_devpts:1506 - bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_allocate_ttys:964 - allocated pty "/dev/pts/0" with master fd 11 and slave fd 14
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_allocate_ttys:964 - allocated pty "/dev/pts/1" with master fd 15 and slave fd 16
      lxc-start 101 20180115032617.725 INFO     lxc_conf - conf.c:lxc_allocate_ttys:984 - finished allocating 2 pts devices
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_setup_ttys:921 - Bind mounted "/dev/pts/0" onto "/dev/tty1"
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:lxc_setup_ttys:921 - Bind mounted "/dev/pts/1" onto "/dev/tty2"
      lxc-start 101 20180115032617.725 INFO     lxc_conf - conf.c:lxc_setup_ttys:930 - Finished setting up 2 /dev/tty<N> device(s)
      lxc-start 101 20180115032617.725 INFO     lxc_conf - conf.c:setup_personality:1542 - set personality to '0x0'
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_caps:2368 - drop capability 'mac_admin' (33)
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_caps:2368 - drop capability 'mac_override' (32)
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_caps:2368 - drop capability 'sys_time' (25)
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_caps:2368 - drop capability 'sys_module' (16)
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_caps:2368 - drop capability 'sys_rawio' (17)
      lxc-start 101 20180115032617.725 DEBUG    lxc_conf - conf.c:setup_caps:2377 - capabilities have been setup
      lxc-start 101 20180115032617.725 NOTICE   lxc_conf - conf.c:lxc_setup:3277 - Container "101" is set up
      lxc-start 101 20180115032617.725 INFO     lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:238 - changed apparmor profile to lxc-container-default-cgns
      lxc-start 101 20180115032617.725 NOTICE   lxc_start - start.c:start:1550 - Exec'ing "/sbin/init".
      lxc-start 101 20180115032617.726 NOTICE   lxc_start - start.c:post_start:1561 - Started "/sbin/init" with pid "29477".
      lxc-start 101 20180115032617.726 NOTICE   lxc_start - start.c:signal_handler:350 - Received SIGCHLD from pid 29365 instead of container init 29477.
      lxc-start 101 20180115032623.781 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_init_pid"
      lxc-start 101 20180115032623.781 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_cgroup"
      lxc-start 101 20180115032623.781 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_state"
      lxc-start 101 20180115032623.781 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_cgroup"
      lxc-start 101 20180115032623.781 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_state"
      lxc-start 101 20180115032623.781 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_init_pid"
      lxc-start 101 20180115032624.905 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_init_pid"
      lxc-start 101 20180115032624.906 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_cgroup"
      lxc-start 101 20180115032624.906 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_state"
      lxc-start 101 20180115032624.906 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_cgroup"
      lxc-start 101 20180115032624.907 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_state"
      lxc-start 101 20180115032624.907 DEBUG    lxc_commands - commands.c:lxc_cmd_handler:1037 - Peer has disconnected for "get_init_pid"
      lxc-start 101 20180115032627.220 DEBUG    lxc_start - start.c:signal_handler:354 - Container init process 29477 exited.
      lxc-start 101 20180115032627.220 DEBUG    lxc_start - start.c:__lxc_start:1502 - Container "101" is rebooting.
      lxc-start 101 20180115032627.220 INFO     lxc_error - error.c:lxc_error_set_and_log:54 - Child <29477> ended on signal (1)
      lxc-start 101 20180115032627.220 INFO     lxc_network - network.c:lxc_delete_network_priv:2534 - Interface "eth0" with index 14 already deleted or existing in different network namespace
      lxc-start 101 20180115032627.220 INFO     lxc_network - network.c:lxc_delete_network_priv:2544 - Removed interface "eth0" with index 14
      lxc-start 101 20180115032627.285 INFO     lxc_network - network.c:lxc_delete_network_priv:2565 - Removed interface "veth101i0" from ""
      lxc-start 101 20180115032627.285 DEBUG    lxc_network - network.c:lxc_delete_network:3129 - Deleted network devices
      lxc-start 101 20180115032627.319 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "101", config section "lxc".
      lxc-start 101 20180115032627.821 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "101", config section "lxc".
      lxc-start 101 20180115032628.915 ERROR    lxc_conf - conf.c:run_buffer:438 - Script exited with status 1.
      lxc-start 101 20180115032628.915 ERROR    lxc_start - start.c:lxc_fini:753 - Failed to run lxc.hook.post-stop for container "101".
      lxc-start 101 20180115032628.915 WARN     lxc_start - start.c:lxc_fini:755 - Container will be stopped instead of rebooted.
    
    



    Logfile after CT 100 was started in the background and 101 was rebooted

      lxc-start 101 20180115032651.690 INFO     lxc_start_ui - tools/lxc_start.c:main:280 - using rcfile /var/lib/lxc/101/config
      lxc-start 101 20180115032651.691 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for reject_force_umount action 0(kill).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for reject_force_umount action 0(kill).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:276 - Setting Seccomp rule to reject force umounts.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .[all].
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .kexec_load errno 1.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for kexec_load action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for kexec_load action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .open_by_handle_at errno 1.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for open_by_handle_at action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for open_by_handle_at action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .init_module errno 1.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for init_module action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for init_module action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .finit_module errno 1.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for finit_module action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for finit_module action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:435 - processing: .delete_module errno 1.
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:610 - Adding native rule for delete_module action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:614 - Adding compat rule for delete_module action 327681(errno).
      lxc-start 101 20180115032651.691 INFO     lxc_seccomp - seccomp.c:parse_config_v2:624 - Merging in the compat Seccomp ctx into the main one.
      lxc-start 101 20180115032651.691 INFO     lxc_conf - conf.c:run_script_argv:457 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "101", config section "lxc".
      lxc-start 101 20180115032652.311 DEBUG    lxc_start - start.c:setup_signal_fd:301 - Set SIGCHLD handler with file descriptor: 5.
      lxc-start 101 20180115032652.349 DEBUG    console - console.c:lxc_console_peer_default:459 - using "/dev/tty" as peer tty device
      lxc-start 101 20180115032652.351 DEBUG    console - console.c:lxc_console_sigwinch_init:151 - process 32093 created signal fd 9 to handle SIGWINCH events
      lxc-start 101 20180115032652.352 DEBUG    console - console.c:lxc_console_winsz:71 - set winsz dstfd:6 cols:125 rows:41
      lxc-start 101 20180115032652.353 INFO     lxc_start - start.c:lxc_init:680 - container "101" is initialized
      lxc-start 101 20180115032652.414 INFO     lxc_conf - conf.c:run_script:507 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "101", config section "net".
      lxc-start 101 20180115032652.308 DEBUG    lxc_network - network.c:instantiate_veth:219 - Instantiated veth "veth101i0/vethKM87M1", index is "16"
      lxc-start 101 20180115032652.308 INFO     lxc_cgroup - cgroups/cgroup.c:cgroup_init:67 - cgroup driver cgroupfs-ng initing for 101
      lxc-start 101 20180115032652.309 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:filter_and_set_cpus:469 - No isolated cpus detected.
      lxc-start 101 20180115032652.309 DEBUG    lxc_cgfsng - cgroups/cgfsng.c:handle_cpuset_hierarchy:640 - "cgroup.clone_children" was already set to "1".
 

Attachments

  • ui.png
    ui.png
    92.6 KB · Views: 16
Log shows that your init process simply dies, so there is something wrong there. Any hint in the container syslog? What debian version do you run inside the container?
 
Just tested and confirmed that the container OS doesn't seem to be a factor, I had been using Debian 9.1-9.3 in the containers but spun up separate CentOS and Arch Linux containers, once one is started, the other cannot start and returns the same result when started with the debug log level.

dmesg within the containers doesn't seem to catch the failed start, and since this isn't a widely-reported issue, i'm almost certain the issue lies in my host config. Is there a way I can further log the init process to find out where it's hanging up at? My host is up to date under the stretch pve-no-subscription branch, and rolling the kernel back to 4.10 does allow me to run all of my containers simultaneously, so I can troubleshoot in that environment as well.

Edit: A fresh install of PVE inside a nested machine functions properly, perhaps I misconfigured something when setting the server up months back.
 
Last edited:
In my case it was trying to add an IP via container /etc/pve/lxc/*.conf to a vmbr bridge that didnt exist. The /etc/network/interfaces of the host had an error and the bridge didnt exist.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!