[SOLVED] NFS File System Mount Problem. Apparmor?

[mattlach]

Hi everyone,

So I've had the following setup working for quite some time.

1.) ZFS pool on host shared via NFS to QEMU VM.
2.) QEMU VM mounts NFS share, reverse encrypts the file system using encfs and exports it via NFS to a Ubuntu Server LXC container.
3.) LXC container runs cloud backup software, which syncs the encfs encrypted directory structure to offsite backup.

My reason for this setup was because I didn't trust the security of th eoffsite backup. (My philosophy is, never trust anything if you aren't the sole holder of the key) So I encrypt it before presenting it to the 3rd party backup software.

So this has worked fine for almost a year, until my recent periodic update cycle when I ran all my updates in my LXC containers, VM's and on the host and rebooted.

Today my offisite backup email notification notified me via email that my file selection for nbackup was down from a large size to 0B.

I started with the LXC container in #3, and confirmed that the encrypted folders were no longer mounted.

So this tells me one of three things happened:
1.) My updates inside my LXC container did something to break it's ability to mount NFS shares
2.) My updates inside my QEMU VM did something to break it's ability to share the NFS folders; or
3.) My updates on the host somehow changed the VM or LXC configurations to stop this from working.


While inside the LXC container, trying to mount them manually by using the mount location defined in /etc/fstab gives me the following error:

$ mount /home/encrypted/data
mount.nfs: access denied by server while mounting xxx.xxx.xxx.xxx:/exports/data

If I do a grep on the IP address in dmesg I find the following:

$ dmesg |grep -i xxx.xxx.xxx.xxx
[437505.126241] audit: type=1400 audit(1483663289.018:180): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/home/encrypted/data" pid=11172 comm="mount.nfs" fstype="nfs" srcname="xxx.xxx.xxx.xxx:/exports/data" flags="rw, nosuid, nodev, noexec"

So this seems to give me conflicting data.

The console command outputs that the server is rejecting the connection, but dmesg is telling me that it is the LXC apparmor configuration that is blocking the mount...

Can anyone suggest what might have changed in my recent host/vm/container system updates that might have broken this previously working configuration?

My guess right now is that I changed something on the host regarding apparmor to allow these shares to mount inside of LXC i the first place when I first set this up, and that when I ran the recent updates on the host, the apparmor config file was overwritten and I lost this configuration, leading to the current situation. Problem is, I can't remember what I did the first time around

I'd appreciate any suggestions that might help me fix this.

Thanks,
Matt

 

[mattlach]

Alright,

So I located the apparmor LXC configuration folder on the host, but it contains four different files, and I'm not sure which one does what.

root@proxmox:/etc/apparmor.d/lxc# ls -l
total 18
-rw-r--r-- 1 root root 558 Apr 16  2016 lxc-default
-rw-r--r-- 1 root root 528 Dec  2 08:39 lxc-default-cgns
-rw-r--r-- 1 root root 544 Dec  2  2015 lxc-default-with-mounting
-rw-r--r-- 1 root root 527 Dec  2 08:39 lxc-default-with-nesting

lxc-default

root@proxmox:/etc/apparmor.d/lxc# cat lxc-default
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,

  # allow nfs mount everywhere
  mount fstype=rpc_pipefs,
  mount fstype=nfs,
}

lxc-default-cgns

root@proxmox:/etc/apparmor.d/lxc# cat lxc-default-cgns
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-cgns flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,
}

lxc-default-with-mounting

root@proxmox:/etc/apparmor.d/lxc# cat lxc-default-with-mounting
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-mounting flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

# allow standard blockdevtypes.
# The concern here is in-kernel superblock parsers bringing down the
# host with bad data.  However, we continue to disallow proc, sys, securityfs,
# etc to nonstandard locations.
  mount fstype=ext*,
  mount fstype=xfs,
  mount fstype=btrfs,
}

lxc-default-with-nesting

root@proxmox:/etc/apparmor.d/lxc# cat lxc-default-with-nesting
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-with-nesting flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>
  #include <abstractions/lxc/start-container>

  deny /dev/.lxc/proc/** rw,
  deny /dev/.lxc/sys/** rw,
  mount fstype=proc -> /var/cache/lxc/**,
  mount fstype=sysfs -> /var/cache/lxc/**,
  mount options=(rw,bind),
  mount fstype=cgroup -> /sys/fs/cgroup/**,
}

Looking at the above, lxc-default was last modified in April 2016, which was when I set this all up.

It contains the lines:
# allow nfs mount everywhere
mount fstype=rpc_pipefs,
mount fstype=nfs,

Which looks like what I might have added to make this work in the first place.

Judging by the error message in the container dmesg in my previous post, it seems like lxc-default-cgns is the file that contains the rule that is blocking what I am trying to do, but I am uncertain what I need to do to change it to make it work.

What is confusing me is the remaining three files, and how the lxc-defualt-cgns file is the one that is blocking me. Did these three recently get added with some update? Are the latter three loaded after the default file, and thus overriding what I added to it?


I think the last time I did this, I added some lines to lxc-default to allow mounting of nfs file systems, but now I am no longer sure where to put these lines (and can't remember exactly what they were) to make this work.

I'd appreciate any help!

--Matt

 

[mattlach]

Well, I added my lines from lxc-default to lxc-default-cgns so now they bot contain the following:

# allow nfs mount everywhere
mount fstype=rpc_pipefs,
mount fstype=nfs,

and then I restarted apparmor as follows:

$ /etc/init.d/apparmor reload

And this seems to be working now. Not quite sure why it happened in the first place, but these files were somehow messed with by a recent update, and this fixed it.

Is there any way I can do this differently so a problem like this doesn't happen again?

Thanks,
Matt

 

[dietmar]

Is there any way I can do this differently so a problem like this doesn't happen again?

This is a feature. NFS mounts are disabled for security reasons.

But we plan to allow such mounts in future by adding feature flags, which can be set by the user more easily ...

 

[mattlach]

This is a feature. NFS mounts are disabled for security reasons.

But we plan to allow such mounts in future by adding feature flags, which can be set by the user more easily ...

Understood, Apparmor may be a royal pain to deal with, but security is important.

I guess my question was, is there any way I can use custom Apparmor configurations without future updates overriding them again?

Did I edit the wrong files? Sometimes there are files that are intended to be edited, and others that are intended to be left alone.

Usually apt warns of existing configuration files that have been changed that would be impacted during an update, and gives you the option to see the differences and choose which one to keep, or at lweast take note of the change and know to go back and fix it, but this did not happen in this case.

 

[fabian]

Understood, Apparmor may be a royal pain to deal with, but security is important.

I guess my question was, is there any way I can use custom Apparmor configurations without future updates overriding them again?

Did I edit the wrong files? Sometimes there are files that are intended to be edited, and others that are intended to be left alone.

Usually apt warns of existing configuration files that have been changed that would be impacted during an update, and gives you the option to see the differences and choose which one to keep, or at lweast take note of the change and know to go back and fix it, but this did not happen in this case.

our LXC packages switched the default profile they use, hence your changes got lost. you can override the used profile (as root) by adding "lxc.aa_profile=YOURCUSTOMPROFILE" to the respective config file in /etc/pve/lxc/ , but you still have the risk of an upstream or PVE change breaking your custom profile (since we naturally don't test with your profile )

 

[jedie]

This is a feature. NFS mounts are disabled for security reasons.

But we plan to allow such mounts in future by adding feature flags, which can be set by the user more easily ...

Is something like this done?

 

[DJFienx]

Is something like this done?

Yep, you can just enable NFS/CIFS mounting from the webui under Options-Feature

 

[strausmann]

Any Update for the Options-Feature ?

 

[Dunuin]

You can enable the NFS/CIFS using the webUI, but only for privileged LXCs. For unprivileged LXCs that still isn't allow (and both features are grayed out in webUI) because of security reasons.