Proxmox 4.1 can not start LXC Container

[Mr.Gosh]

My Proxmox VE 4.1 Server can not start newly created LXC Containers.

What is going wrong here?

lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 346 To get more details, run the container in foreground mode.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.
TASK ERROR: command 'lxc-start -n 126' failed: exit code 1

complete pvereport is too long so only the main things interesting for lxc are in here:

Spoiler: stripped pvereport

==== general system info ====
# hostname
pve00
# pveversion --verbose
proxmox-ve: 4.1-41 (running kernel: 4.2.8-1-pve)
pve-manager: 4.1-22 (running version: 4.1-22/aca130cf)
pve-kernel-4.2.6-1-pve: 4.2.6-36
pve-kernel-4.2.8-1-pve: 4.2.8-41
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 1.0-1
pve-cluster: 4.0-36
qemu-server: 4.0-64
pve-firmware: 1.1-7
libpve-common-perl: 4.0-54
libpve-access-control: 4.0-13
libpve-storage-perl: 4.0-45
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-9
pve-container: 1.0-52
pve-firewall: 2.0-22
pve-ha-manager: 1.0-25
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-7
lxcfs: 2.0.0-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.1.9 pve00.router.local pve00 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
# top -b -n 1 | head -n 15
top - 13:26:37 up 17 days, 19:20, 1 user, load average: 0.26, 0.17, 0.22
Tasks: 368 total, 1 running, 367 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.6 us, 0.2 sy, 0.0 ni, 99.0 id, 0.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65926764 total, 20404632 used, 45522132 free, 317516 buffers
KiB Swap: 33554428 total, 0 used, 33554428 free. 2141712 cached Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1411 root 20 0 9258540 7.556g 13648 S 6.4 12.0 658:40.86 kvm
20410 root 20 0 3089904 2.029g 13720 S 6.4 3.2 819:39.28 kvm
26859 root 20 0 980736 310892 13640 S 6.4 0.5 90:32.69 kvm
32398 root 20 0 25884 2964 2380 R 6.4 0.0 0:00.01 top
1 root 20 0 29568 5716 3128 S 0.0 0.0 0:28.90 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.15 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:13.09 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H

# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
Stepping: 4
CPU MHz: 2399.988
CPU max MHz: 2600.0000
CPU min MHz: 1200.0000
BogoMIPS: 4200.35
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 15360K
NUMA node0 CPU(s): 0-11

==== info about storage (lvm and zfs) ====
# cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
maxfiles 0
content rootdir,vztmpl,images,iso

nfs: ISOs
export /mnt/tank1/public/Isos
path /mnt/pve/ISOs
server 192.168.1.10
content iso,vztmpl
maxfiles 1
options vers=3

nfs: container
export /mnt/tank1/nfs-containers
path /mnt/pve/container
server 192.168.1.10
content images,rootdir
maxfiles 1
options vers=3

# cat /etc/pve/lxc/125.conf
arch: amd64
cpulimit: 4
cpuunits: 1024
hostname: kibana01
memory: 4096
net0: bridge=vmbr0,gw=192.168.1.1,hwaddr=66:39:32:64:30:64,ip=192.168.1.45/24,name=eth0,type=veth
ostype: ubuntu
rootfs: local:125/vm-125-disk-1.raw,size=80G
swap: 512

# cat /etc/pve/lxc/126.conf
arch: amd64
cpulimit: 4
cpuunits: 1024
hostname: kibana02
memory: 4096
net0: bridge=vmbr0,gw=192.168.1.1,hwaddr=3A:30:37:38:63:64,ip=192.168.1.45/24,name=eth0,type=veth
ostype: ubuntu
rootfs: container:126/vm-126-disk-2.raw,size=8G
swap: 512


==== info about network and firewall ====
# ifconfig
eth0 Link encap:Ethernet HWaddr ac:22:0b:8a:81:e4
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:481739164 errors:0 dropped:51282 overruns:0 frame:0
TX packets:86143515 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:643207369542 (599.0 GiB) TX bytes:67000677480 (62.3 GiB)
Interrupt:16 Memory:fb300000-fb320000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:738031 errors:0 dropped:0 overruns:0 frame:0
TX packets:738031 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:438987527 (418.6 MiB) TX bytes:438987527 (418.6 MiB)

vmbr0 Link encap:Ethernet HWaddr ac:22:0b:8a:81:e4
inet addr:192.168.1.9 Bcast:192.168.1.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:91618828 errors:0 dropped:1628 overruns:0 frame:0
TX packets:46778200 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:610235996795 (568.3 GiB) TX bytes:59917714786 (55.8 GiB)

# cat /etc/network/interfaces
auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet static
address 192.168.1.9
netmask 255.255.255.0
gateway 192.168.1.1
bridge_ports eth0
bridge_stp off
bridge_fd 0
# iptables-save
# Generated by iptables-save v1.4.21 on Tue Apr 5 13:26:39 2016
*filter
:INPUT ACCEPT [86190070:610344322215]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [47012518:59683114931]
COMMIT
# Completed on Tue Apr 5 13:26:39 2016

==== info about bios ====
# dmidecode -t bios
# dmidecode 2.12
SMBIOS 2.7 present.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
Vendor: American Megatrends Inc.
Version: 5507
Release Date: 07/19/2015
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 8192 kB
Characteristics:
PCI is supported
APM is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
8042 keyboard services are supported (int 9h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
ACPI is supported
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 4.6

Handle 0x0079, DMI type 13, 22 bytes
BIOS Language Information
Language Description Format: Long
Installable Languages: 1
en|US|iso8859-1
Currently Installed Language: en|US|iso8859-1

 

[Ovidiu]

I had a similar issue and one of the moderators gave me the advice to start the container in the foreground with logging like this, that might give you more insights. Start it like this, once its up stop it, then check the log file.

lxc-start -n 126 -F -lDEBUG -o /path/to/log/file

 

[Mr.Gosh]

I had a similar issue and one of the moderators gave me the advice to start the container in the foreground with logging like this, that might give you more insights. Start it like this, once its up stop it, then check the log file.

lxc-start -n 126 -F -lDEBUG -o /path/to/log/file

thanks for the hint - it seems to be related to the network creation:

lxc-start: conf.c: instantiate_veth: 2767 failed to create veth pair (veth126i0 and veth80ITPG): Operation not supported 
lxc-start: conf.c: lxc_create_network: 3084 failed to create netdev 
lxc-start: start.c: lxc_spawn: 954 failed to create the network 
lxc-start: start.c: __lxc_start: 1211 failed to spawn '126' 
lxc-start: lxc_start.c: main: 344 The container failed to start. 
lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the --logfile and --logpriority options.

but until now - I didn't find a solution for the problem... :-(

 

[aychprox]

Try to create new CT with quota option disable, it should good to go.

 

[Mr.Gosh]

Try to create new CT with quota option disable, it should good to go.

the quota option was diasabled - I verified that by creating a new Container and double checked if the quota option is not checked - still the same.

But when I delete the network card - than the system boots! Re-Adding produces the same error again.
So it defiantly has something todo with the network stack creation...

 

[Mr.Gosh]

does anyone have the same problem?

 

[t.lamprecht]

If you made a recent kernel upgrade please reboot your node, this may fix this issue.

 

[Emilio González Montaña]

THANKS A LOT!!! It worked (restart after kernel upgrade), I just want going nuts