[SOLVED] LXC won't start after upgrade to 5.1-38

N

Neonsigns

New Member
Oct 27, 2017
17
0
1
47
Just upgraded Proxmox to 5.1-38 from the no-subscription repo. Now none of my existing containers or any new containers will start. They seem to get stuck on starting the eth0 network interface and fails to start. Please advise.

Log is attached. Thanks
 

Attachments

Could you please provide some more information, including the container's configuration on PVE side as well as what distribution/template & settings you have inside?
 
Here is the pve config.

lxc.arch = amd64
lxc.include = /usr/share/lxc/config/ubuntu.common.conf
lxc.monitor.unshare = 1
lxc.tty.max = 2
lxc.environment = TERM=linux
lxc.uts.name = TEST
lxc.cgroup.memory.limit_in_bytes = 536870912
lxc.cgroup.memory.memsw.limit_in_bytes = 1073741824
lxc.cgroup.cpu.shares = 1024
lxc.rootfs.path = /var/lib/lxc/104/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth104i0
lxc.net.0.hwaddr = 96:15:D5:9A:90:42
lxc.net.0.name = eth0

Distribution is Ubuntu 16.04

Firewall is disabled

Thanks
 
The lxc config doesn't seem unusual.
The pve config is usually more useful though (`/etc/pve/lxc/104.conf` or `# pct config 104`).

Also the output of lxc-start run in foreground (`# lxc-start -F -n 104`) can often be more helpful than the logs from lxc as it might contain stderr output of the processes involved.
 
pve lxc config:

arch: amd64
cores: 1
hostname: TEST
memory: 512
net0: name=eth0,bridge=vmbr0,hwaddr=96:15:D5:9A:90:42,ip=dhcp,ip6=auto,type=veth
ostype: ubuntu
rootfs: local-cntstorage:subvol-104-disk-1,size=10G
swap: 512

Lxc start foreground is attached. Thanks
 

Attachments

I am facing the same situation in my lab. I am already facing it few days. It's looks like some interference between AppArmor and LXC.
Putting Apparmor in complain mode help to avoid LXC container blocking:
sudo aa-complain lxc-container-default-cgns
sudo aa-complain /etc/apparmor.d/lxc/lxc-default-cgns

I have tried to fiddle with APP profiles without a any success.

Does not look like PVE problem.


Looks like it happened after that:

Start-Date: 2017-12-03 18:37:02
Commandline: apt full-upgrade
Requested-By: ......... (1000)
Upgrade: linux-libc-dev:amd64 (4.9.51-1, 4.9.65-1), libc6-dev:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libc6:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libdbi1:amd64 (0.9.0-4+b2, 0.9.0-4+deb9u1), locales:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libapparmor1:amd64 (2.11.0-3, 2.11.0-3+deb9u1), libc-l10n:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libapparmor-perl:amd64 (2.11.0-3, 2.11.0-3+deb9u1), libc-bin:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libc-dev-bin:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), multiarch-support:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), apparmor:amd64 (2.11.0-3, 2.11.0-3+deb9u1)
End-Date: 2017-12-03 18:37:10


Dec 7 20:08:51 vgmps-kvm00 kernel: [91591.578543] audit: type=1400 audit(1512670131.025:432): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=13837 comm="lxc-start" flags="rw, rslave"

Dec 7 20:08:51 vgmps-kvm00 kernel: [91592.275819] audit: type=1400 audit(1512670131.723:433): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/var/lib/lxc/100/rootfs/" pid=13847 comm="mount" fstype="ext4" srcname="/dev/loop0"

Dec 7 20:08:52 vgmps-kvm00 kernel: [91592.972358] audit: type=1400 audit(1512670132.419:435): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/usr/lib/x86_64-linux-gnu/lxc/rootfs/" pid=13942 comm="lxc-start" srcname="/var/lib/lxc/100/rootfs/" flags="rw, rbind"

Dec 7 20:08:52 vgmps-kvm00 kernel: [91592.995914] audit: type=1400 audit(1512670132.443:437): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/" pid=13942 comm="lxc-start" fstype="tmpfs" srcname="none"

Dec 7 20:08:52 vgmps-kvm00 kernel: [91592.995968] audit: type=1400 audit(1512670132.443:438): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/" pid=13942 comm="lxc-start" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"

Dec 7 20:08:52 vgmps-kvm00 kernel: [91592.995978] audit: type=1400 audit(1512670132.443:439): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/tty/" pid=13942 comm="lxc-start" srcname="/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/net/" flags="rw, bind"

Dec 7 20:08:52 vgmps-kvm00 kernel: [91592.996053] audit: type=1400 audit(1512670132.443:440): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/" pid=13942 comm="lxc-start" srcname="/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/" flags="rw, bind"

Dec 7 20:08:52 vgmps-kvm00 kernel: [91592.996057] audit: type=1400 audit(1512670132.443:441): apparmor="ALLOWED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/" pid=13942 comm="lxc-start" flags="ro, nosuid, nodev, noexec, remount, bind"


sudo

root@vgmps-kvm00:~$ sudo aa-status
apparmor module is loaded.
6 profiles are loaded.
4 profiles are in enforce mode.
/usr/sbin/ntpd
lxc-container-default
lxc-container-default-with-mounting
lxc-container-default-with-nesting
2 profiles are in complain mode.
/usr/bin/lxc-start
lxc-container-default-cgns

14 processes have profiles defined.
0 processes are in enforce mode.

14 processes are in complain mode.
/usr/bin/lxc-start (13837)
lxc-container-default-cgns (13942)
lxc-container-default-cgns (14019)
lxc-container-default-cgns (14057)
lxc-container-default-cgns (14058)
lxc-container-default-cgns (14064)
lxc-container-default-cgns (14065)
lxc-container-default-cgns (14085)
lxc-container-default-cgns (14139)
lxc-container-default-cgns (14146)
lxc-container-default-cgns (14147)
lxc-container-default-cgns (14300)
lxc-container-default-cgns (14301)
lxc-container-default-cgns (14302)
0 processes are unconfined but have a profile defined.
 
Vaidotas said:
I am facing the same situation in my lab. I am already facing it few days. It's looks like some interference between AppArmor and LXC.
Putting Apparmor in complain mode help to avoid LXC container blocking:
sudo aa-complain lxc-container-default-cgns
sudo aa-complain /etc/apparmor.d/lxc/lxc-default-cgns

I have tried to fiddle with APP profiles without a any success.

Does not look like PVE problem.


Looks like it happened after that:

Start-Date: 2017-12-03 18:37:02
Commandline: apt full-upgrade
Requested-By: ......... (1000)
Upgrade: linux-libc-dev:amd64 (4.9.51-1, 4.9.65-1), libc6-dev:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libc6:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libdbi1:amd64 (0.9.0-4+b2, 0.9.0-4+deb9u1), locales:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libapparmor1:amd64 (2.11.0-3, 2.11.0-3+deb9u1), libc-l10n:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libapparmor-perl:amd64 (2.11.0-3, 2.11.0-3+deb9u1), libc-bin:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), libc-dev-bin:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), multiarch-support:amd64 (2.24-11+deb9u1, 2.24-11+deb9u2), apparmor:amd64 (2.11.0-3, 2.11.0-3+deb9u1)
End-Date: 2017-12-03 18:37:10
Click to expand...

please downgrade to apparmor packages in version 2.11.0-3. 2.11.0-3+deb9u1 was a proposed upgrade for Debian 9.3 which has been retracted since (exactly because of breakage like this). don't run production systems with stretch-proposed-upgrades enabled ;)

see https://forum.proxmox.com/threads/pct-start-failed-after-apparmor-update.38479/#post-190519 for details
 
I have the same issue with a LXC Container after upgrading to PVE 5.1-41 (non-subscriber)
after apt-get upgrade in a ubuntu 16.04 lxc the container doesn't boot up anymore and stays on an empty prompt. Restoring Backup fixes the issue and issue is reproduceable.

apparmor,2.11.0-3
libapparmor-perl,2.11.0-3
libapparmor1,2.11.0-3
 
You must log in or register to reply here.
Top Bottom