[SOLVED] NFS File System Mount Problem. Apparmor?

Discussion in 'Proxmox VE: Networking and Firewall' started by mattlach, Jan 6, 2017.

  1. mattlach

    mattlach Member

    Joined:
    Mar 23, 2016
    Messages:
    145
    Likes Received:
    12
    Hi everyone,

    So I've had the following setup working for quite some time.

    1.) ZFS pool on host shared via NFS to QEMU VM.
    2.) QEMU VM mounts NFS share, reverse encrypts the file system using encfs and exports it via NFS to a Ubuntu Server LXC container.
    3.) LXC container runs cloud backup software, which syncs the encfs encrypted directory structure to offsite backup.

    My reason for this setup was because I didn't trust the security of th eoffsite backup. (My philosophy is, never trust anything if you aren't the sole holder of the key) So I encrypt it before presenting it to the 3rd party backup software.

    So this has worked fine for almost a year, until my recent periodic update cycle when I ran all my updates in my LXC containers, VM's and on the host and rebooted.

    Today my offisite backup email notification notified me via email that my file selection for nbackup was down from a large size to 0B.

    I started with the LXC container in #3, and confirmed that the encrypted folders were no longer mounted.

    So this tells me one of three things happened:
    1.) My updates inside my LXC container did something to break it's ability to mount NFS shares
    2.) My updates inside my QEMU VM did something to break it's ability to share the NFS folders; or
    3.) My updates on the host somehow changed the VM or LXC configurations to stop this from working.


    While inside the LXC container, trying to mount them manually by using the mount location defined in /etc/fstab gives me the following error:

    Code:
    $ mount /home/encrypted/data
    mount.nfs: access denied by server while mounting xxx.xxx.xxx.xxx:/exports/data
    If I do a grep on the IP address in dmesg I find the following:

    Code:
    $ dmesg |grep -i xxx.xxx.xxx.xxx
    [437505.126241] audit: type=1400 audit(1483663289.018:180): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/home/encrypted/data" pid=11172 comm="mount.nfs" fstype="nfs" srcname="xxx.xxx.xxx.xxx:/exports/data" flags="rw, nosuid, nodev, noexec"
    So this seems to give me conflicting data.

    The console command outputs that the server is rejecting the connection, but dmesg is telling me that it is the LXC apparmor configuration that is blocking the mount...

    Can anyone suggest what might have changed in my recent host/vm/container system updates that might have broken this previously working configuration?

    My guess right now is that I changed something on the host regarding apparmor to allow these shares to mount inside of LXC i the first place when I first set this up, and that when I ran the recent updates on the host, the apparmor config file was overwritten and I lost this configuration, leading to the current situation. Problem is, I can't remember what I did the first time around :p

    I'd appreciate any suggestions that might help me fix this.

    Thanks,
    Matt
     
  2. mattlach

    mattlach Member

    Joined:
    Mar 23, 2016
    Messages:
    145
    Likes Received:
    12
    Alright,

    So I located the apparmor LXC configuration folder on the host, but it contains four different files, and I'm not sure which one does what.

    Code:
    root@proxmox:/etc/apparmor.d/lxc# ls -l
    total 18
    -rw-r--r-- 1 root root 558 Apr 16  2016 lxc-default
    -rw-r--r-- 1 root root 528 Dec  2 08:39 lxc-default-cgns
    -rw-r--r-- 1 root root 544 Dec  2  2015 lxc-default-with-mounting
    -rw-r--r-- 1 root root 527 Dec  2 08:39 lxc-default-with-nesting
    lxc-default
    Code:
    root@proxmox:/etc/apparmor.d/lxc# cat lxc-default
    # Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
    # will source all profiles under /etc/apparmor.d/lxc
    
    profile lxc-container-default flags=(attach_disconnected,mediate_deleted) {
      #include <abstractions/lxc/container-base>
    
      # the container may never be allowed to mount devpts.  If it does, it
      # will remount the host's devpts.  We could allow it to do it with
      # the newinstance option (but, right now, we don't).
      deny mount fstype=devpts,
    
      # allow nfs mount everywhere
      mount fstype=rpc_pipefs,
      mount fstype=nfs,
    }
    lxc-default-cgns
    Code:
    root@proxmox:/etc/apparmor.d/lxc# cat lxc-default-cgns
    # Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
    # will source all profiles under /etc/apparmor.d/lxc
    
    profile lxc-container-default-cgns flags=(attach_disconnected,mediate_deleted) {
      #include <abstractions/lxc/container-base>
    
      # the container may never be allowed to mount devpts.  If it does, it
      # will remount the host's devpts.  We could allow it to do it with
      # the newinstance option (but, right now, we don't).
      deny mount fstype=devpts,
      mount fstype=cgroup -> /sys/fs/cgroup/**,
    }
    lxc-default-with-mounting
    Code:
    root@proxmox:/etc/apparmor.d/lxc# cat lxc-default-with-mounting
    # Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
    # will source all profiles under /etc/apparmor.d/lxc
    
    profile lxc-container-default-with-mounting flags=(attach_disconnected,mediate_deleted) {
      #include <abstractions/lxc/container-base>
    
    # allow standard blockdevtypes.
    # The concern here is in-kernel superblock parsers bringing down the
    # host with bad data.  However, we continue to disallow proc, sys, securityfs,
    # etc to nonstandard locations.
      mount fstype=ext*,
      mount fstype=xfs,
      mount fstype=btrfs,
    }
    lxc-default-with-nesting
    Code:
    root@proxmox:/etc/apparmor.d/lxc# cat lxc-default-with-nesting
    # Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
    # will source all profiles under /etc/apparmor.d/lxc
    
    profile lxc-container-default-with-nesting flags=(attach_disconnected,mediate_deleted) {
      #include <abstractions/lxc/container-base>
      #include <abstractions/lxc/start-container>
    
      deny /dev/.lxc/proc/** rw,
      deny /dev/.lxc/sys/** rw,
      mount fstype=proc -> /var/cache/lxc/**,
      mount fstype=sysfs -> /var/cache/lxc/**,
      mount options=(rw,bind),
      mount fstype=cgroup -> /sys/fs/cgroup/**,
    }
    
    Looking at the above, lxc-default was last modified in April 2016, which was when I set this all up.

    It contains the lines:
    # allow nfs mount everywhere
    mount fstype=rpc_pipefs,
    mount fstype=nfs,

    Which looks like what I might have added to make this work in the first place.

    Judging by the error message in the container dmesg in my previous post, it seems like lxc-default-cgns is the file that contains the rule that is blocking what I am trying to do, but I am uncertain what I need to do to change it to make it work.

    What is confusing me is the remaining three files, and how the lxc-defualt-cgns file is the one that is blocking me. Did these three recently get added with some update? Are the latter three loaded after the default file, and thus overriding what I added to it?


    I think the last time I did this, I added some lines to lxc-default to allow mounting of nfs file systems, but now I am no longer sure where to put these lines (and can't remember exactly what they were) to make this work.

    I'd appreciate any help!

    --Matt
     
    #2 mattlach, Jan 6, 2017
    Last edited: Jan 6, 2017
  3. mattlach

    mattlach Member

    Joined:
    Mar 23, 2016
    Messages:
    145
    Likes Received:
    12
    Well, I added my lines from lxc-default to lxc-default-cgns so now they bot contain the following:

    # allow nfs mount everywhere
    mount fstype=rpc_pipefs,
    mount fstype=nfs,

    and then I restarted apparmor as follows:

    $ /etc/init.d/apparmor reload

    And this seems to be working now. Not quite sure why it happened in the first place, but these files were somehow messed with by a recent update, and this fixed it.

    Is there any way I can do this differently so a problem like this doesn't happen again?

    Thanks,
    Matt
     
  4. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,279
    Likes Received:
    279
    This is a feature. NFS mounts are disabled for security reasons.

    But we plan to allow such mounts in future by adding feature flags, which can be set by the user more easily ...
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. mattlach

    mattlach Member

    Joined:
    Mar 23, 2016
    Messages:
    145
    Likes Received:
    12
    Understood, Apparmor may be a royal pain to deal with, but security is important.

    I guess my question was, is there any way I can use custom Apparmor configurations without future updates overriding them again?

    Did I edit the wrong files? Sometimes there are files that are intended to be edited, and others that are intended to be left alone.

    Usually apt warns of existing configuration files that have been changed that would be impacted during an update, and gives you the option to see the differences and choose which one to keep, or at lweast take note of the change and know to go back and fix it, but this did not happen in this case.
     
  6. fabian

    fabian Proxmox Staff Member
    Staff Member

    Joined:
    Jan 7, 2016
    Messages:
    3,129
    Likes Received:
    479
    our LXC packages switched the default profile they use, hence your changes got lost. you can override the used profile (as root) by adding "lxc.aa_profile=YOURCUSTOMPROFILE" to the respective config file in /etc/pve/lxc/ , but you still have the risk of an upstream or PVE change breaking your custom profile (since we naturally don't test with your profile ;))
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice