Apparmor in privileged container

Trevize

New Member
Apr 12, 2022
1
0
1
I have a problem which might be normal behavior or not, I'm looking for confirmation.
Every time I start a privileged container or restart apparmor inside I get the following message in the host's syslog:

Code:
Apr 12 17:49:12 pm kernel: [154462.321869] audit: type=1400 audit(1649778552.937:390): apparmor="STATUS" operation="profile_replace" info="not policy admin" error=-13 label="lxc-115_</var/lib/lxc>//&:lxc-115_<-var-lib-lxc>:unconfined" pid=4082008 comm="apparmor_parser"

Also apparmor inside the container fails:

Code:
* apparmor.service - Load AppArmor profiles
     Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Tue 2022-04-12 16:03:06 UTC; 15s ago
       Docs: man:apparmor(7)
             https://gitlab.com/apparmor/apparmor/wikis/home/
    Process: 77 ExecStart=/lib/apparmor/apparmor.systemd reload (code=exited, status=1/FAILURE)
   Main PID: 77 (code=exited, status=1/FAILURE)
        CPU: 1.058s

Apr 12 16:03:05 ptest apparmor.systemd[128]: /sbin/apparmor_parser: Unable to replace "nvidia_modprobe".  Permission denied; attempted to load a profile wh>
Apr 12 16:03:06 ptest apparmor.systemd[130]: /sbin/apparmor_parser: Unable to replace "/usr/lib/NetworkManager/nm-dhcp-client.action".  Permission denied; >
Apr 12 16:03:06 ptest apparmor.systemd[132]: /sbin/apparmor_parser: Unable to replace "/usr/bin/man".  Permission denied; attempted to load a profile while>
Apr 12 16:03:06 ptest apparmor.systemd[134]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Apr 12 16:03:06 ptest apparmor.systemd[136]: /sbin/apparmor_parser: Unable to replace "/usr/sbin/tcpdump".  Permission denied; attempted to load a profile >
Apr 12 16:03:06 ptest apparmor.systemd[77]: Error: At least one profile failed to load
Apr 12 16:03:06 ptest systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Apr 12 16:03:06 ptest systemd[1]: apparmor.service: Failed with result 'exit-code'.
Apr 12 16:03:06 ptest systemd[1]: Failed to start Load AppArmor profiles.
Apr 12 16:03:06 ptest systemd[1]: apparmor.service: Consumed 1.058s CPU time.

Even so everything works this really grinds my gears.
Running unprivileged container with or without nesting does not produce this error message. Enabling nesting on the privileged container after creation does not change anything, I still get the error message.

(I'm running privileged containers because I'm bind mounting ZFS datasets inside the container and I want to see the same user/groups inside and outside. It's a home server on a local network)

The template is base Ubuntu 20.04 downloaded via the GUI.

pveversion -v

Code:
proxmox-ve: 7.1-1 (running kernel: 5.13.19-4-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-helper: 7.1-10
pve-kernel-5.13: 7.1-7
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-7
pve-kernel-libc-dev: 5.15.19-1
pve-kernel-5.13.19-4-pve: 5.13.19-9
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-2
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-1
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-5
pve-cluster: 7.1-3
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-5
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-1
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1
 
hello,

I also have this message with ubuntu CT 21.10 (privileged container)
seems doesn't have any error with CT debian 11 (privileged container)
 
Last edited:
Also happening for me on Ubuntu 22.04 template.
I've resorted to using the Debian 11 template as apparmor does not appear to be enabled, but it is not ideal.
 
Last edited:
hello,

I also have this message with ubuntu CT 21.10 (privileged container)
seems doesn't have any error with CT debian 11 (privileged container)
I suspect because CT Debian 11 does not have apparmor enabled. Try running aa-status or aa-enabled at the commandline to confirm. I get the following.

Bash:
apparmor module is loaded.
apparmor filesystem is not mounted.
 
Last edited:
Also happening for me on Ubuntu 22.04 template.
I've resorted to using the Debian 11 template as apparmor does not appear to be enabled, but it is not ideal.
I get a variety of other services failing in the Debian 11 template. I don't want to resort to running a privileged container without AppArmor; my understanding is that I might as well just be running it on the host at that point.
 
I have this issue as well. Anyone find a solution yet? I'm trying to run docker in and LXC with Ubuntu 20.04. Tried nesting and non nested both, same issue.
 
@Prox Mox, is anyone able to provide some insight into this issue? This is a real blocker, especially with Privileged Containers, but seems to affect Unprivileged ones alike.

In my case, it's affecting several services, but the one I'm most concerned with is ntp — see below:
Code:
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  ntp-doc
The following NEW packages will be installed:
  ntp
0 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 721 kB of archives.
After this operation, 2121 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 ntp amd64 1:4.2.8p15+dfsg-1ubuntu2 [721 kB]
Fetched 721 kB in 0s (4507 kB/s)
Selecting previously unselected package ntp.
(Reading database ... 34125 files and directories currently installed.)
Preparing to unpack .../ntp_1%3a4.2.8p15+dfsg-1ubuntu2_amd64.deb ...
Unpacking ntp (1:4.2.8p15+dfsg-1ubuntu2) ...
Setting up ntp (1:4.2.8p15+dfsg-1ubuntu2) ...
Created symlink /etc/systemd/system/network-pre.target.wants/ntp-systemd-netif.path -> /lib/systemd/system/ntp-systemd-netif.path.
Created symlink /etc/systemd/system/multi-user.target.wants/ntp.service -> /lib/systemd/system/ntp.service.
apparmor_parser: Unable to replace "/usr/sbin/ntpd".  Permission denied; attempted to load a profile while confined?
ntp-systemd-netif.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.10.2-1) ...
root@dc1:~# systemctl enable ntp-systemd-netif.service
The unit files have no installation config (WantedBy=, RequiredBy=, Also=,
Alias= settings in the [Install] section, and DefaultInstance= for template
units). This means they are not meant to be enabled using systemctl.
 
Possible reasons for having this kind of units are:
* A unit may be statically enabled by being symlinked from another unit's
  .wants/ or .requires/ directory.
* A unit's purpose may be to act as a helper for some other unit which has
  a requirement dependency on it.
* A unit may be started when needed via activation (socket, path, timer,
  D-Bus, udev, scripted systemctl call, ...).
* In case of template units, the unit is meant to be enabled with some
  instance name specified.
 
Last edited:
I've resorted to putting everything in Debian 11 LXC/Dockers. Maybe at some point we can move back to Ubuntu as an option.
 
@dietmar or @Matthias. , any ideas here? TL;DR: We're trying to create LXC containers (priv and unpriv alike) with the Proxmox-supplied templates, and we're getting AppArmor failures, resulting in the AppArmor service refusing to start. Different results between Debian vs Ubuntu, different results (sometimes?) with Privileged vs Unprivileged. I wouldn't normally ping, but this has been happening for a while. Hope I haven't missed a bigger thread or something, apologies if I have, or if you are the wrong folks to ask!

Brand new LXC with Unbuntu 22.04 template:
Nov 06 20:14:56 newlxc apparmor.systemd[109]: /sbin/apparmor_parser: Unable to replace "kmod". Permission denied; attempted to load a profile while confined?

If i'm not mistaken, AppArmor used to run without issue within an LXC container - is this something we've done wrong on our end (host config or something), or is something actually broken?
 
Must be that this was simply not considered an error?
Do you really need to load apparmor profiles inside your container?

Administrating apparmor profiles requires CAP_MAC_ADMIN which is dropped by the default common lxc configuration.

You can add a config snippet to enable this:

Code:
# /usr/share/lxc/config/common.conf.d/02-stacked-apparmor.conf
# Clear this (as the main common.conf fills it with the capabilities below plus mac_admin and mac_override
lxc.cap.drop =

# Drop some harmful capabilities
lxc.cap.drop = sys_time sys_module sys_rawio

Since we do have apparmor stacking/nesting available now, this should be mostly safe (but then since you're using privileged containers, safety isn't really a thing anyway).

We could probably automate this based on the availability of stacking though. (/sys/kernel/security/apparmor/features/domain/stack must contain yes)
 
Must be that this was simply not considered an error?
Do you really need to load apparmor profiles inside your container?

Administrating apparmor profiles requires CAP_MAC_ADMIN which is dropped by the default common lxc configuration.

You can add a config snippet to enable this:

Code:
# /usr/share/lxc/config/common.conf.d/02-stacked-apparmor.conf
# Clear this (as the main common.conf fills it with the capabilities below plus mac_admin and mac_override
lxc.cap.drop =

# Drop some harmful capabilities
lxc.cap.drop = sys_time sys_module sys_rawio

Since we do have apparmor stacking/nesting available now, this should be mostly safe (but then since you're using privileged containers, safety isn't really a thing anyway).

We could probably automate this based on the availability of stacking though. (/sys/kernel/security/apparmor/features/domain/stack must contain yes)

Hey, thanks for the quick response!

Truthfully, it seems I might be a bit ignorant about the subject; I just know that the AppArmor service did not used to fail and now it does; but it sounds like what you're saying is that AppArmor running on the host is enough? I guess I need to brush up a bit more on how containers interact with the host OS.

I'm personally not doing anything like Docker within LXC at the moment where I'd *need* it. So it seems I can just safely ignore this as a non-issue for my current use case? (My container is just an SMB server for a home network).

This is good information for others who need it though, so thanks again for the informative post.
 
Must be that this was simply not considered an error?
Do you really need to load apparmor profiles inside your container?

Administrating apparmor profiles requires CAP_MAC_ADMIN which is dropped by the default common lxc configuration.

You can add a config snippet to enable this:

Code:
# /usr/share/lxc/config/common.conf.d/02-stacked-apparmor.conf
# Clear this (as the main common.conf fills it with the capabilities below plus mac_admin and mac_override
lxc.cap.drop =

# Drop some harmful capabilities
lxc.cap.drop = sys_time sys_module sys_rawio

Since we do have apparmor stacking/nesting available now, this should be mostly safe (but then since you're using privileged containers, safety isn't really a thing anyway).

We could probably automate this based on the availability of stacking though. (/sys/kernel/security/apparmor/features/domain/stack must contain yes)
Hello there,
I am sorry I need to be enlightened.
Actually what I have is that :

#PBS:/usr/share/lxc/config/common.conf.d# ls 00-lxcfs.conf 01-pve.conf README
Should I create 02-stacked-apparmor.conf ? If so then, I have to paste : lxc.cap.drop = sys_time sys_module sys_rawio
Right?
Thanks
 
Last edited:
Hello there,
I am sorry I need to be enlightened.
Actually what I have is that :

#PBS:/usr/share/lxc/config/common.conf.d# ls 00-lxcfs.conf 01-pve.conf README
Should I create 02-stacked-apparmor.conf ? If so then, I have to paste : lxc.cap.drop = sys_time sys_module sys_rawio
Right?
Thanks
I'm in the same boat, did you get anywhere with this?
 
Hello,
I have a similar issue with installing and running services with docker in a ubuntu 22.04 container in my Proxmox server node.
Apologies because I am new to Linux environment. Just started getting into it :)

I am able to install and create docker containers and run services successfully in a VM but when I try to do the same in a LXC Container it fails.
If I install and create docker containers at the Node level it works perfectly but doing the same in the container doesn't work properly.

In my case I have been trying to run Salcorn with a docker-compose.yml file; some services are successfully installed but its not able to successfully create the db. I keep getting this error.
Bash:
Creating saltcorn-db ... error

ERROR: for saltcorn-db  Cannot start service saltcorn-db: AppArmor enabled on system but the docker-default profile could not be loaded: running `/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default531342309` failed with output: apparmor_parser: Unable to replace "docker-default".  Permission denied; attempted to load a profile while confined?

error: exit status 243

ERROR: for saltcorn-db  Cannot start service saltcorn-db: AppArmor enabled on system but the docker-default profile could not be loaded: running `/sbin/apparmor_parser apparmor_parser -Kr /var/lib/docker/tmp/docker-default531342309` failed with output: apparmor_parser: Unable to replace "docker-default".  Permission denied; attempted to load a profile while confined?

error: exit status 243
ERROR: Encountered errors while bringing up the project.

Update: also tried to do the same thing on a debian 11 (not having apparmor) container with the below error message:
Bash:
Creating saltcorn-db ... error

ERROR: for saltcorn-db  Cannot start service saltcorn-db: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown

ERROR: for saltcorn-db  Cannot start service saltcorn-db: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: process_linux.go:458: setting cgroup config for procHooks process caused: can't load program: operation not permitted: unknown
ERROR: Encountered errors while bringing up the project.

I have also disabled apparmor service on the container and rebooted to no success. Tried also to disable apparmor on the Node and rebooted still no success.

I am assuming that docker-default profile is automatically added to apparmor once docker is installed. It seems so on the Node and any VM but when it comes to the container it doesn't load the profile automatically. I have tried this in Priviliged and Unprivileged containers but in both modes its unsuccessful.

I have no idea how to manually create docker-default profile in apparmor and give permissions for my use case.

If anyone can please guide me to the right path or provide a solution I would be grateful :D


Proxmox server node:
Bash:
cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Bash:
16 profiles are loaded.
16 profiles are in enforce mode.
   /usr/bin/lxc-start
   /usr/bin/man
   /usr/sbin/chronyd
   docker-default
   lsb_release
   lxc-102_</var/lib/lxc>
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   man_filter
   man_groff
   nvidia_modprobe
   nvidia_modprobe//kmod
   swtpm
   tcpdump
0 profiles are in complain mode.
26 processes have profiles defined.
26 processes are in enforce mode.
   /usr/bin/lxc-start (68243)
   /usr/sbin/chronyd (790)
   /usr/sbin/chronyd (804)
   /usr/lib/systemd/systemd (68267) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/sbin/cron (68591) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/bin/dbus-daemon (deleted) (68592) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /lib/systemd/systemd-logind (deleted) (68596) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /sbin/agetty (deleted) (68606) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /bin/login (deleted) (68607) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /sbin/agetty (deleted) (68608) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /lib/systemd/systemd (deleted) (69040) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /lib/systemd/systemd (deleted) (69041) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /bin/bash (deleted) (69045) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/sbin/rsyslogd (74649) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/sbin/uuidd (75662) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/postfix/sbin/master (80125) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/postfix/sbin/pickup (80127) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/postfix/sbin/qmgr (80128) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/accountsservice/accounts-daemon (80473) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/bin/python3.8 (80507) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/sbin/sshd (80784) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/systemd/systemd-networkd (80827) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/systemd/systemd-resolved (80837) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/lib/systemd/systemd-journald (80841) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/bin/containerd (94443) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
   /usr/bin/dockerd (94663) lxc-102_</var/lib/lxc>//&:lxc-102_<-var-lib-lxc>:unconfined
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

LXC Container created in the node: (Priviliged Container)
Bash:
cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Bash:
aa-status
apparmor module is loaded.
0 profiles are loaded.
0 profiles are in enforce mode.
0 profiles are in complain mode.
0 processes have profiles defined.
0 processes are in enforce mode.
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.
 
Last edited:
Docker does not run on this privileged LXC container of Ubuntu 22.04 LTS, probably due to this apparmor issue. Please help to fix it, thanks a lot.
 
any update on this?
I can also use another version of ubuntu...which one is supported?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!