[SOLVED] Apparmor preventing LXCs starting after update

[bradgy]

I ran an apt update && apt dist-upgrade on my home server after a few weeks of uptime as part of its routine maintenance but its LXCs are failing to start after the reboot. All the VMs are still working.

journalctl -xe output:

-- Unit pve-container@200.service has begun starting up.
Mar 10 13:57:03 bjserver audit[1951]: AVC apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=1951 comm="lxc-start" flags="rw, rslave"
Mar 10 13:57:03 bjserver kernel: audit: type=1400 audit(1520708223.230:19): apparmor="DENIED" operation="mount" info="failed mntpnt match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=1951 comm="lxc-start
Mar 10 13:57:03 bjserver lxc-start[1949]: lxc-start: 200: lxccontainer.c: wait_on_daemonized_start: 751 No such file or directory - Failed to receive the container state
Mar 10 13:57:03 bjserver lxc-start[1949]: lxc-start: 200: tools/lxc_start.c: main: 368 The container failed to start.
Mar 10 13:57:03 bjserver lxc-start[1949]: lxc-start: 200: tools/lxc_start.c: main: 370 To get more details, run the container in foreground mode.
Mar 10 13:57:03 bjserver lxc-start[1949]: lxc-start: 200: tools/lxc_start.c: main: 372 Additional information can be obtained by setting the --logfile and --logpriority options.
Mar 10 13:57:03 bjserver systemd[1]: pve-container@200.service: Control process exited, code=exited status=1
Mar 10 13:57:03 bjserver systemd[1]: pve-container@200.service: Killing process 1951 (lxc-start) with signal SIGKILL.
Mar 10 13:57:03 bjserver systemd[1]: Failed to start PVE LXC Container: 200.
-- Subject: Unit pve-container@200.service has failed
-- Defined-By: systemd
-- Support: (url)
--
-- Unit pve-container@200.service has failed.
--
-- The result is failed.
Mar 10 13:57:03 bjserver systemd[1]: pve-container@200.service: Unit entered failed state.
Mar 10 13:57:03 bjserver systemd[1]: pve-container@200.service: Failed with result 'exit-code'.
Mar 10 13:57:03 bjserver pvedaemon[1946]: command 'systemctl start pve-container@200' failed: exit code 1
Mar 10 13:57:03 bjserver pvedaemon[1273]: <root@pam> end task UPID:bjserver:0000079A:0000409C:5AA42A7F:vzstart:200:root@pam: command 'systemctl start pve-container@200' failed: exit code 1
Mar 10 13:57:03 bjserver pve-guests[1293]: <root@pam> end task UPID:bjserver:00000510:00000441:5AA429E4:startall::root@pam: OK

I'm not super knowledgeable about containers, just enough to get myself into trouble, so any ideas on fixes are appreciated.

 

[r33int]

+1, I have encountered the same issue, as well as a friend.

As a temporary fix, you can downgrade apparmor to 2.10.95-4 from backports:
https: //packages.debian.org/jessie-backports/apparmor

Download the appropriate package for your system

Install it using:

dpkg -i filename.deb

And hold it, to make sure it does not get upgraded with an apt upgrade:

apt-mark hold apparmor

If you want to upgrade it, unhold it:

apt-mark unhold apparmor

Now, start apparmor

systemctl start apparmor

Hope this will work for you as well!

 

[bradgy]

Hot dang.
This worked!
I'd seen other posts talking about downgrading the apparmor package but they were from late last year so I naively thought that they wouldn't apply any more.
Thanks for making a post r33int! You are a lifesaver.

 

[r33int]

Hot dang.
This worked!
I'd seen other posts talking about downgrading the apparmor package but they were from late last year so I naively thought that they wouldn't apply any more.
Thanks for making a post r33int! You are a lifesaver.

Glad I helped you!

 

[fabian]

please don't install packages from jessie-backports on PVE 5 systems!

the AppArmor issue has been solved in an lxc-pve update last week, so unless you have incorrect PVE repositories configured it should not be possible to hit it. PVE 4 is not affected at all, since the relevant AppArmor change was only for Debian Stretch.

 

[tjh]

I guess neither of these two have either
a) a proper subscription or
b) like me, use the pve-no-subscription repo.

If you only upgraded using the default apt entries (And you did not have a valid sub), it would be possible to get into this situation I guess?

 

[fabian]

I guess neither of these two have either
a) a proper subscription or
b) like me, use the pve-no-subscription repo.

If you only upgraded using the default apt entries (And you did not have a valid sub), it would be possible to get into this situation I guess?

yes, if you don't have PVE repositories configured which you can access, you won't get PVE updates.

 

[philipborg]

yes, if you don't have PVE repositories configured which you can access, you won't get PVE updates.

I just installed a fresh 5.1-3 version of Proxmox and at first it worked. Then I updated with a simple apt-get update and apt-get full-upgrade and I got this error and could not start any LXC containers. (Never tried KVM) I needed to manually add
deb http://download.proxmox.com/debian/pve stretch pve-no-subscription then run apt-get full-upgrade (and update ofcourse) to fix it. I am writing this as a heads up to all other people who has the same problem and as sort of bug report.

 

[bradgy]

Thanks for the info Fabian and others. I've learnt something valuable here. I thought I took things pretty slowly setting everything up but there's always more documentation to read and understand; in this case, setting up repositories correctly. I'll look into a subscription as well in order to support the work the Proxmox team does.

edit: for anyone reading this later... setting up the pve-no-subscription repo as described here solved my issue with apparmor.

 

[Jota V.]

But, it's safe add pve-no-subscription in production servers ? Can i patch this bug without add this repo?

 

[MrBMT]

But, it's safe add pve-no-subscription in production servers ? Can i patch this bug without add this repo?

Proxmox do not recommend using the pve-no-subscription repository for production servers, as packages are not always heavily tested/validated.

The subscription repository is recommended for commercial/production servers as the packages provided there are tested much more.

Realistically, most of the time you're probably going to be okay using the pve-no-subscription repository; however if you're using the server for commercial purposes, I'd recommend paying for the subscription repo personally.

 

[Jota V.]

Proxmox do not recommend using the pve-no-subscription repository for production servers, as packages are not always heavily tested/validated.

The subscription repository is recommended for commercial/production servers as the packages provided there are tested much more.

Realistically, most of the time you're probably going to be okay using the pve-no-subscription repository; however if you're using the server for commercial purposes, I'd recommend paying for the subscription repo personally.

Thank you!

 

[Deleted member 55253]

What's the path forward for non-subscribers? I hit this bug last night, thankfully stumbled upon this post for a quick fix. My cluster's not used for anything commercial / "production" per se, just a homelab, so paying the subscription fee for my three processors isn't ideal.

I haven't added or removed any repos, just using the stock ones that come with 5.1-35.

 

[fabian]

What's the path forward for non-subscribers? I hit this bug last night, thankfully stumbled upon this post for a quick fix. My cluster's not used for anything commercial / "production" per se, just a homelab, so paying the subscription fee for my three processors isn't ideal.

I haven't added or removed any repos, just using the stock ones that come with 5.1-35.

if you don't have a subscription, you need to enable the no-subscription repository like described in the docs. otherwise, you don't get updates, and thus also no bug and security fixes.