Please help me understand these apparmor log entries

[Ovidiu]

Everything seems to work fine but saw these log entries:

Apr 01 09:53:02 james kernel:  [44540.247389] audit: type=1400 audit(1459497182.260:29): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/pstore/" pid=3790 comm="mount" fstype="pstore" srcname="pstore" flags="rw"
Apr 01 09:53:02 james kernel:  [44540.249481] audit: type=1400 audit(1459497182.260:30): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/pstore/" pid=3790 comm="mount" fstype="pstore" srcname="pstore" flags="ro"
Apr 01 09:53:02 alfred kernel:  [44540.418302] audit: type=1400 audit(1459497182.428:31): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=4034 comm="mount" flags="rw, remount, silent"
Apr 01 09:53:02 alfred kernel:  [44540.421142] audit: type=1400 audit(1459497182.432:32): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=4035 comm="mount" flags="rw, remount, relatime"
Apr 01 09:53:02 james kernel:  [44540.446470] audit: type=1400 audit(1459497182.456:34): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/lock/" pid=4101 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 01 09:53:02 alfred kernel:  [44540.441307] audit: type=1400 audit(1459497182.452:33): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/" pid=4090 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
Apr 01 09:53:02 james kernel:  [44540.452029] audit: type=1400 audit(1459497182.464:35): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/proc/" pid=4108 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 01 09:53:02 alfred kernel:  [44540.457336] audit: type=1400 audit(1459497182.468:36): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/" pid=4116 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
Apr 01 09:53:02 bailey kernel:  [44540.479448] audit: type=1400 audit(1459497182.492:37): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=4182 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 01 09:53:02 bailey kernel:  [44540.484752] audit: type=1400 audit(1459497182.496:38): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/dev/pts/" pid=4190 comm="mount" flags="rw, nosuid, noexec, remount, relatime"

james being the host and alfred and bailey the guests.
alfred is unconfined so why is this still happening?

#cat /etc/pve/lxc/100.conf
arch: amd64
cpulimit: 4
cpuunits: 1024
hostname: alfred.ict-consult.co.za
memory: 16384
nameserver: 94.23.250.79 8.8.8.8
net0: bridge=vmbr0,gw=94.23.250.254,hwaddr=02:00:00:71:02:12,ip=51.254.252.80/28,ip6=dhcp,name=eth0,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: local:100/vm-100-disk-1.raw,size=128G,acl=0,quota=1
searchdomain: ict-consult.co.za
startup: order=1,up=120
swap: 0
lxc.aa_profile: unconfined

bailey is unconfined and specifically allows mounting

#cat /etc/pve/lxc/102.conf
arch: amd64
cpulimit: 2
cpuunits: 1024
hostname: bailey.ict-consult.co.za
memory: 2048
nameserver: 94.23.250.79 8.8.8.8
net0: bridge=vmbr0,gw=94.23.250.254,hwaddr=02:00:00:1e:cf:c3,ip=51.254.252.82/28,ip6=dhcp,name=eth0,type=veth
onboot: 1
ostype: debian
protection: 1
rootfs: local:102/vm-102-disk-1.raw,size=32G,acl=0
searchdomain: ict-consult.co.za
startup: order=2,up=30
swap: 0
lxc.aa_profile: unconfined
lxc.mount.entry: /dev/fuse dev/fuse none bind,create=file 0 0

Also, why are the logs showing: lxc-container-default for both containers?

 

[fabian]

Did you restart your containers after adding the lxc.aa_profile line?

Also, I am not sure what you mean by "specifically allows mounting"? lxc.mount.entry does not allow mounting, but tells lxc to mount something on startup (and can lead to a lot of problems compared to our mpX config options, because we don't care about lxc.mount.entry at all!)

 

[Ovidiu]

Ok, sorry about that, I clearly misunderstood what the lxc.mount.entry means then :-(

Yes, I restarted the containers just yesterday evening and that line has been in there for months.

 

[fabian]

Can you post the contents of /var/lib/lxc/ID/config for those containers?

 

[Ovidiu]

guest alfred = ID 100

#cat /var/lib/lxc/100/config
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.monitor.unshare = 1
lxc.tty = 2
lxc.environment = TERM=linux
lxc.utsname = alfred.mydomain.co.za
lxc.cgroup.memory.limit_in_bytes = 17179869184
lxc.cgroup.memory.memsw.limit_in_bytes = 17179869184
lxc.cgroup.cpu.cfs_period_us = 100000
lxc.cgroup.cpu.cfs_quota_us = 400000
lxc.cgroup.cpu.shares = 1024
lxc.rootfs = /var/lib/lxc/100/rootfs
lxc.network.type = veth
lxc.network.veth.pair = veth100i0
lxc.network.hwaddr = 02:00:00:71:02:12
lxc.network.name = eth0
lxc.aa_profile = unconfined

guest bailey = ID 102

#cat /var/lib/lxc/102/config
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.monitor.unshare = 1
lxc.tty = 2
lxc.environment = TERM=linux
lxc.utsname = bailey.mydomain.co.za
lxc.cgroup.memory.limit_in_bytes = 2147483648
lxc.cgroup.memory.memsw.limit_in_bytes = 2147483648
lxc.cgroup.cpu.cfs_period_us = 100000
lxc.cgroup.cpu.cfs_quota_us = 200000
lxc.cgroup.cpu.shares = 1024
lxc.rootfs = /var/lib/lxc/102/rootfs
lxc.network.type = veth
lxc.network.veth.pair = veth102i0
lxc.network.hwaddr = 02:00:00:1e:cf:c3
lxc.network.name = eth0
lxc.aa_profile = unconfined
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file 0 0

 

[fabian]

works as expected here (even with your exact configs).. did you modify any of the lxc config templates in /usr/share/lxc/config ? are you running uptodate versions of the pve packages?

 

[Ovidiu]

nope, never touched anything in /usr/share/lxc :-(
Everything is up-to-date as of yesterday evening.

 

[fabian]

You can try manually starting the containers in foreground mode with debug logging and see if there are hints why the apparmor profile is not set correctly (replace ID with your container ID and /path/to/log/file with a path where you want to store the log file):

lxc-start -n ID -F -lDEBUG -o /path/to/log/file

After the container has started up, you can shut it down using "pct shutdown ID"

 

[Ovidiu]

OK, my container 100 is a live machine but I can play at will with ID 102 so here it goes:

#lxc-start -n 102 -F -lDEBUG -o /root/102.log
readline() on closed filehandle $fd at /usr/share/lxc/hooks/lxc-pve-autodev-hook line 29.
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
[info] Not setting System Clock.
[ ok ] Activating swap...done.
[warn] Fast boot enabled, so skipping file system check. ... (warning).
[ ok ] Cleaning up temporary files... /tmp.
[ ok ] Mounting local filesystems...done.
[ ok ] Activating swapfile swap...done.
[ ok ] Cleaning up temporary files....
[ ok ] Setting kernel variables ...done.
[....] Configuring network interfaces...Failed to bring up eth0.
done.
[ ok ] Starting rpcbind daemon....
[ ok ] Starting NFS common utilities: statd idmapd.
[ ok ] Cleaning up temporary files....
[ ok ] Setting up X socket directories... /tmp/.X11-unix /tmp/.ICE-unix.
[ ok ] Setting sensors limits.
INIT: Entering runlevel: 2
[info] Using makefile-style concurrent boot in runlevel 2.
[ ok ] Starting enhanced syslogd: rsyslogd.
Sandstorm started. PID = 1122
[ ok ] Starting deferred execution scheduler: atd.
[ ok ] Starting periodic command scheduler: cron.
[ ok ] Starting system message bus: dbus.
[ ok ] Starting OpenBSD Secure Shell server: sshd.
Starting NIXStats Agent
[ ok ] Starting Postfix Mail Transport Agent: postfix.
[....] Starting SNMP services::  snmpd

and its been hanging there for 15 minutes now. The container is up though and working. I have attached the full log file as I couldn't see anything confidential inside.

I opened a second terminal and sent: pct shutdown 102 which resulted in the following output in the first terminal where I had started the container:

INIT: Sending processes the TERM signal
[info] Using makefile-style concurrent boot in runlevel 0.
[....] Stopping SNMP services::  snmpdWaiting for PID 1122 to terminate...
Sandstorm server stopped.
[ ok ] Stopping Postfix Mail Transport Agent: postfix.
[ ok ] Stopping deferred execution scheduler: atd.
Stopping NIXStats Agent
[ ok ] Asking all remaining processes to terminate...done.
[ ok ] All processes ended within 1 seconds...done.
[ ok ] Stopping enhanced syslogd: rsyslogd.
[ ok ] Stopping rpcbind daemon....
[ ok ] Deconfiguring network interfaces...done.
[ ok ] Stopping NFS common utilities: idmapd statd.
[info] Not saving System Clock.
[ ok ] Deactivating swap...done.
[....] Unmounting local filesystems...umount: /dev/fuse: block devices are not permitted on filesystem
failed.
mount: / is busy
[info] Will now halt.
[6:506] 09:56 [root@james] ~

###edit###
Just adding a note that I do seem to have SNMP problems, I use a LXC container containing Observium to monitor all guests as well as the host and about 2 days ago it simply stopped working and debugging it I saw there are no SNMP connections possible to the guests/host anymore. Was going to open a separate thread about that but it seems its connected.

 

[fabian]

Note: when you run an lxc container in foreground mode (using -F) and it starts without errors, it does not "hang", but runs in the foreground until you shut it down this is perfectly normal!

You log shows that lxc switches the profile to unconfined:

  lxc-start 1459755339.262 INFO  lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:187 - changed apparmor profile to unconfined

I would suggest removing the fuse mount point entry (FUSE and LXC don't mix), and updating your installation if the problem persists. I don't see anything wrong with your configuration or logs (besides the fuse entry).

 

[Ovidiu]

Thanks for helping out Fabian.

Unfortunately it seems I need the fuse: https://forum.proxmox.com/threads/i...appliance-for-sandstorm-io.24773/#post-124178 or is there meanwhile another solution?

And this being the last line:

[....] Starting SNMP services::  snmpd

doesn't mean it was hanging but just that it was started last? I'm asking as I am having snmp connectivity problems but will open a new thread if you can confirm this looks all OK.

P.S. What do you mean by: updating your installation? Everything is up-to-date.

 

[fabian]

Thanks for helping out Fabian.

Unfortunately it seems I need the fuse: https://forum.proxmox.com/threads/i...appliance-for-sandstorm-io.24773/#post-124178 or is there meanwhile another solution?

According to the sandstorm issue tracker they don't need FUSE anymore: https://github.com/sandstorm-io/sandstorm/issues/858 , so you might try removing that line to get a bit closer to the recommended setup (running unconfined is not recommended either..). if you leave it in, don't try to use the snapshot backup feature or lxc-freeze, they will most likely break if anything uses FUSE from within the container.

And this being the last line:

[....] Starting SNMP services::  snmpd

doesn't mean it was hanging but just that it was started last? I'm asking as I am having snmp connectivity problems but will open a new thread if you can confirm this looks all OK.

Yes, the output just "stops" when it's finished booting. In your case, the last service does not display OK, so this might or might not point to an issue with that specific service (have you checked the logs of that service in particular?). Feel free to open a new thread for a specific issue with the snmpd service.

P.S. What do you mean by: updating your installation? Everything is up-to-date.

That's good! I was just making sure since you did not post any version information.

 

[Ovidiu]

Thanks, I tried removing the fuse and unconfined settings but sandstorm gives this error and exits. If you don't have any advice, I'll move this issue over to the sandstorm forums.

** Starting Sandstorm at: Mon Apr  4 14:48:59 2016
*** Uncaught exception ***
sandstorm/run-bundle.c++:1077: failed: mount("none", "/", nullptr, MS_REC | MS_PRIVATE, nullptr): Permission denied
stack: 0x4f0ab3 0x4efff1 0x509dc0 0x509129 0x50911a 0x4a84cf 0x4a7fea
** Server monitor died. Aborting.

 

[fabian]

You probably need the unconfined (or another, less restricted) app armor profile (at least as far as I understand what sandstorm is and does ). So my suggestion was to remove FUSE, but leave the unconfined profile in for now (but be aware that the container will not be restricted very much afterwards, the app armor containment is there for a reason).

 

[Ovidiu]

just for the sake of completeness, I had to reboot yesterday as I changed IP and MAC addresses for alfred and saw similar entries again. Wondering why it is still saying: lxc-container-default :-(

Apr 15 17:50:07 alfred kernel:  [1017957.725753] audit: type=1400 audit(1460735407.574:109): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=8414 comm="mount" flags="rw, remount, relatime"
Apr 15 17:50:07 alfred kernel:  [1017957.820267] audit: type=1400 audit(1460735407.666:111): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/lock/" pid=8480 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 15 17:50:07 alfred kernel:  [1017957.824473] audit: type=1400 audit(1460735407.670:112): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/proc/" pid=8487 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 15 17:50:07 alfred kernel:  [1017957.851309] audit: type=1400 audit(1460735407.698:114): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=8561 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 15 22:31:53 alfred kernel:  [1034871.285918] audit: type=1400 audit(1460752313.329:118): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=9467 comm="mount" flags="ro, remount, relatime"
Apr 15 22:32:13 alfred kernel:  [1034891.347226] audit: type=1400 audit(1460752333.381:124): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/lock/" pid=10526 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 15 22:32:13 alfred kernel:  [1034891.378877] audit: type=1400 audit(1460752333.413:127): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/run/shm/" pid=10607 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
Apr 15 22:32:37 alfred kernel:  [1034915.503216] audit: type=1400 audit(1460752357.529:129): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default" name="/" pid=13326 comm="mount" flags="ro, remount, relatime"

all the info for alfred is still the same as above. I don'T really expect any solution, just wanted to complete the thread with all info, in case I need to research it again at a later time.

 

[sahostking]

I get the same thing. Quite strange. :

Jun 04 21:50:01 vz-cpt-2 kernel: audit: type=1400 audit(1465069801.307:2453): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/home/virtfs/compon/bin/" pid=3147 comm="jailshell" srcname="/bin/" flags="rw, bind"
Jun 04 21:50:01 vz-cpt-2 kernel: audit: type=1400 audit(1465069801.307:2454): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/home/virtfs/compon/bin/" pid=3147 comm="jailshell" flags="ro, nosuid, remount, bind"
Jun 04 21:50:01 vz-cpt-2 kernel: audit: type=1400 audit(1465069801.307:2455): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/home/virtfs/compon/bin/" pid=3147 comm="jailshell" flags="rw, nosuid, remount, bind"
Jun 04 21:50:01 vz-cpt-2 kernel: audit: type=1400 audit(1465069801.307:2456): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/home/virtfs/compon/dev/" pid=3147 comm="jailshell" srcname="/dev/" flags="rw, bind"