1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Process hiding in LXC (hidepid proc mount option)

Discussion in 'Proxmox VE: Installation and configuration' started by onlime, Aug 27, 2016.

  1. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    How can I set proc (/proc) mount options for an LXC container on ProxmoxVE 4.2 (using ZFS storage)? I am talking about hidepid=2, which means: Users are only able too see their own processes (like with hidepid=1), but also the other process IDs are hidden for them in /proc.

    On Proxmox 3.4 this was possible for OpenVZ containers simply by adding the following line to /etc/fstab inside the container:

    Code:
    proc /proc proc defaults,noexec,nosuid,nodev,hidepid=2 0 0
    
    This won't work in a LXC container...

    Code:
    $ mount | grep proc
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
    proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
    lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    
     
  2. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    besides, setting hidepid=2 on the ProxmoxVE host node has no effect on the LXC containers either.

    Code:
    hn$ mount -o remount,hidepid=2 /proc
    hn$ mount | grep proc
    proc on /proc type proc (rw,relatime,hidepid=2)
    
    On the host node itself, this works just fine.
     
  3. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    /proc inside a container is its own mount point.
    In order to be able to remount as you suggested you'd need to allow this operation in the apparmor profile like this:
    Code:
    # /etc/apparmor.d/lxc/lxc-default-cgns-with-proc-remount
    profile lxc-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
      #include <abstractions/lxc/container-base>
    
      # these are copied from lxc-container-default-cgns:
      deny mount fstype=devpts,
      mount fstype=cgroup -> /sys/fs/cgroup/**,
    
      # This will allow remounting /proc, eg to change hidepid
      mount options=(rw, nosuid, nodev, noexec, remount, silent, relatime) -> /proc/,
    }
    
    The hidepid option doesn't need to be included there as it's not known by apparmor.

    Reload via
    Code:
     # apparmor_parser -r -W -T /etc/apparmor.d/lxc-containers
    Set the profile for your container in /etc/pve/lxc/$VMID.conf:
    Code:
    (...)
    lxc.aa_profile = lxc-default-cgns-with-proc-remount
     
  4. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    Thanks @wbumiller for these detailed instructions. Couldn't get it running yet, though.

    Reloading the new AppArmor profile with apparmor_parser did not report any errors.
    Tried it with the following in /etc/pve/lxc/$VMID.conf (which is the correct syntax?):

    Code:
    lxc.aa_profile: lxc-default-cgns-with-proc-remount
    # or ...
    lxc.aa_profile = lxc-default-cgns-with-proc-remount
    
    After restarting the LXC container, I tried:

    Code:
    $ mount | grep proc
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    (...)
    
    $ mount -o remount,hidepid=2 /proc
    mount: cannot remount block device proc read-write, is write-protected
    
    Code:
    hn$ dmesg | grep apparmor
    [337202.692601] audit: type=1400 audit(1472502879.621:202): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-default-cgns-with-proc-remount" name="/proc/" pid=9081 comm="mount" flags="rw, nosuid, nodev, noexec, remount"
    
    Once I get this working, will I be able to auto-mount /proc via /etc/fstab as proposed?:

    Code:
    # <file system> <mount point>  <type>  <options>  <dump>  <pass>
    proc     /proc     proc   defaults,noexec,nosuid,nodev,hidepid=2   0   0
    
    Or do you suggest to remount it via init script?
     
    #4 onlime, Aug 29, 2016
    Last edited: Aug 29, 2016
  5. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    Ah, your container isn't using the 'silent' and 'relatime' options required by the profile line I posted. (I just copied what I used in my test container). You'll have to match the `options=(flags)' portion of the AppArmor rule to your container (or add multiple rules for different possible combinations). Should be `options=(rw, nosuid, nodev, noexec, remount) according to the DENIED message output there and the mount flags seen above in your first post.
     
  6. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    Great, thanks! Now it works.

    Remounting worked with the following command:

    Code:
    $ mount -o remount,relatime,hidepid=2 /proc
    
    I had to add the `relatime` flag as it was there before and as I put it into the AppArmor profile as required option. Don't quite get it, why I need to specify it explicitly on the remount command.
    Here's my working solution...

    hn:/etc/apparmor.d/lxc/lxc-default-cgns-with-proc-remount

    Code:
    # Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
    # will source all profiles under /etc/apparmor.d/lxc
    
    profile lxc-container-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
      #include <abstractions/lxc/container-base>
    
      # the container may never be allowed to mount devpts.  If it does, it
      # will remount the host's devpts.  We could allow it to do it with
      # the newinstance option (but, right now, we don't).
      deny mount fstype=devpts,
      mount fstype=cgroup -> /sys/fs/cgroup/**,
    
      # This will allow remounting /proc, eg to change hidepid
      mount options=(rw, nosuid, nodev, noexec, remount, relatime) -> /proc/,
    }
    
    hn:/etc/pve/lxc/$VMID.conf

    Code:
    (...)
    lxc.aa_profile: lxc-container-default-cgns-with-proc-remount
    
    reload AppArmor profiles:

    Code:
    hn$ apparmor_parser -r -W -T /etc/apparmor.d/lxc-containers
    
    ct:/etc/fstab

    Code:
    # <file system> <mount point>  <type>  <options>  <dump>  <pass>
    proc  /proc  proc  defaults,noexec,nosuid,nodev,relatime,hidepid=2 0  0
    
    restart LXC container:

    Code:
    hn$ pct stop $VMID && pct start $VMID
    
    /proc will then be correctly mounted with the `hidepid=2` option. No remounting required!

    Code:
    ct$ mount | grep 'proc on'
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime,hidepid=2)
    proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime,hidepid=2)
    proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime,hidepid=2)
    proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime,hidepid=2)
    
     
    #6 onlime, Aug 30, 2016
    Last edited: Aug 30, 2016
  7. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    I am still getting the following DENIED apparmor messages upon container start:

    Code:
    [376680.591775] audit: type=1400 audit(1472542357.517:395): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/" pid=8574 comm="mount" flags="rw, remount"
    [376680.594481] audit: type=1400 audit(1472542357.521:396): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/" pid=8575 comm="mount" flags="rw, remount, noatime"
    [376680.797361] audit: type=1400 audit(1472542357.725:397): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/run/" pid=8816 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
    [376680.812879] audit: type=1400 audit(1472542357.741:398): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/run/lock/" pid=8833 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [376680.833557] audit: type=1400 audit(1472542357.761:399): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/sys/" pid=8854 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
    [376680.986368] audit: type=1400 audit(1472542357.913:400): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/run/shm/" pid=9018 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [376680.998105] audit: type=1400 audit(1472542357.925:401): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/dev/pts/" pid=9029 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
    
    This also happens on an fresh LXC container with the default lxc-container-default-cgns AppArmor profile:

    Code:
    [376624.608565] audit: type=1400 audit(1472542301.537:387): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=5212 comm="mount" flags="rw, remount"
    [376624.610775] audit: type=1400 audit(1472542301.537:388): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=5213 comm="mount" flags="rw, remount"
    [376624.845170] audit: type=1400 audit(1472542301.773:389): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/" pid=5481 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
    [376624.859926] audit: type=1400 audit(1472542301.785:390): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/lock/" pid=5498 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [376624.870490] audit: type=1400 audit(1472542301.797:391): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/proc/" pid=5508 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [376624.882287] audit: type=1400 audit(1472542301.809:392): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/sys/" pid=5519 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
    [376625.033363] audit: type=1400 audit(1472542301.961:393): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/shm/" pid=5683 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [376625.045081] audit: type=1400 audit(1472542301.973:394): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/dev/pts/" pid=5694 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
    
    Should I report this as a new issue?
     
  8. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    What kind of container is it? And is it converted from openvz or fresh?
     
  9. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    It was a container which was converted from OpenVZ. Basically I went this direct migration path (migration via `vzdump` and `pct restore` definitely is no option for us and I am going to propose the following as alternative migration path):

    1. Stopped all containers (actually replicated them via ZREP to secondary host node and running them there, still on ProxmoxVE 3.4/OpenVZ)
    2. Upgraded primary host node from ProxmoxVE 3.4 to 4.2
    3. Create CT with same VMID via ProxmoxVE WebUI, choosing any template for the new LXC container
    4. moving the existing ZFS volume in place:
    Code:
    $ VMID=184
    $ zfs destroy rpool/zfsdisks/subvol-$VMID-disk-1
    $ zfs rename rpool/ROOT/pve-$VMID rpool/zfsdisks/subvol-$VMID-disk-1
    $ zfs set mountpoint=/rpool/zfsdisks/subvol-$VMID-disk-1 rpool/zfsdisks/subvol-$VMID-disk-1
    
    $ pct start $VMID
    
    This worked fine. When I check your http://pve.proxmox.com/wiki/Convert_OpenVZ_to_LXC#Step_by_step_conversion instructions it looks like nothing needs to be changed inside the container filesystem.

    I have also just set up a fresh LXC container via the ProxmoxVE WebUI and with a fresh basic system image:

    Code:
    $ pveam update
    $ pveam available
    $ pveam download local debian-8.0-standard_8.4-1_amd64.tar.gz
    
    Once I start this new container, I'll get the same apparmor DENIED messages:

    Code:
    $ pct start $VMID
    $ dmesg | grep apparmor
    [380948.425404] audit: type=1400 audit(1472546625.352:424): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/sys/fs/pstore/" pid=26644 comm="mount" fstype="pstore" srcname="pstore"
    [380948.425466] audit: type=1400 audit(1472546625.352:425): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/sys/fs/pstore/" pid=26644 comm="mount" fstype="pstore" srcname="pstore" flags="ro"
    [380948.557403] audit: type=1400 audit(1472546625.484:426): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=26753 comm="mount" flags="rw, remount, silent"
    [380948.559915] audit: type=1400 audit(1472546625.484:427): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=26754 comm="mount" flags="rw, remount, noatime"
    [380948.620682] audit: type=1400 audit(1472546625.548:428): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/" pid=26809 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
    [380948.631870] audit: type=1400 audit(1472546625.556:429): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/lock/" pid=26820 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [380948.639589] audit: type=1400 audit(1472546625.564:430): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/proc/" pid=26827 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [380948.649852] audit: type=1400 audit(1472546625.576:431): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/sys/" pid=26835 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
    [380948.719792] audit: type=1400 audit(1472546625.644:432): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/shm/" pid=26901 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
    [380948.729537] audit: type=1400 audit(1472546625.656:433): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/dev/pts/" pid=26909 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
    
     
  10. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    Ah yes, they're expected in some guest OS types. Some file systems get mounted by lxc before starting up the guest system, which is then not allowed to replace the existing mount points. Not all guest systems produce the same warnings, though. There are fewer such messages in container-aware guests (specifically ones using a new-enough systemd).
     
  11. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    1,982
    Likes Received:
    120
    Could you please describe what sysrq-trigger and relatime for proc inside a container should do? Or is this still some leftovers from OpenVZ?
     
  12. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    Good question! I was also quite confused about these new defaults. I am comparing a plain Debian Jessie OpenVZ container on ProxmoxVE 3.4:

    Code:
    $ mount | grep proc
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    
    vs. a plain Debian Jessie LXC container on ProxmoxVE 4.2:

    Code:
    $ mount | grep proc
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
    proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
    lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    
    `relatime` was already there in OpenVZ, even on the `/proc` mount of the host node. AFAIK, this was always the default. Don't ask me if that makes sense on a proc filesystem...

    Concerning the `/proc/sysrq-trigger` and all the other proc mounts, I would really appreciate if someone could shed some light.
     
  13. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    /proc/sys contains system settings containers should not have write access to, so lxc mounts it as read-only
    /proc/sys/net contains the network part of /proc/sys, which is namespaced and thus can (and should) be read-write (in order to change interface settings such as forwarding, or disabling/enabling ipv6, autoconfiguration, rp-filtering, ...)
    /proc/sysrq-trigger can do lots of nasty stuff[1] and is therefore double protected via a read-only mount and an AppArmor rule.

    I suppose OpenVZ dealt with that via kernel changes, but I haven't checked.

    [1] https://www.kernel.org/doc/Documentation/sysrq.txt
     
  14. LnxBil

    LnxBil Well-Known Member

    Joined:
    Feb 21, 2015
    Messages:
    1,982
    Likes Received:
    120
    Thank you Wolfgang. I was not aware of the fact that /proc is actually mounted without "shielding" important stuff like sysrq automatically. I cannot think of a use case in where it makes sense to have sysrq inside a container, so it should be hidden by the kernel itself.
     
  15. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    Except the kernel doesn't have a concept of containers as such. It just provides building blocks such as namespaces and cgroups, which can also be used independently and for different purposes. I frequently use network namespaces via `ip netns` for tunnels/vpns or stuff I want to isolate only from the network, or mount namespaces (via `unshare`) to hide certain mounts from other unprivileged prying eyes.
    So to the kernel, you're just mounting yet another /proc.
     
    #15 wbumiller, Aug 31, 2016
    Last edited: Aug 31, 2016
  16. onlime

    onlime Member
    Proxmox VE Subscriber

    Joined:
    Aug 9, 2013
    Messages:
    37
    Likes Received:
    7
    Thanks for the explanation @wbumiller
    The following might be a bit off-topic, but I would finally like to understand the default mounts inside an LXC container...

    We now run into another weird issue. If we create a new LXC container from the debian-8.0-standard_8.4-1_amd64.tar.gz template, we see the following mounts:

    Code:
    ct185$ mount
    rpool/zfsdisks/subvol-185-disk-1 on / type zfs (rw,noatime,xattr,posixacl)
    none on /dev type tmpfs (rw,relatime,size=492k,mode=755)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
    proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
    sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
    sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
    fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
    lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
    devpts on /dev/tty1 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
    devpts on /dev/tty2 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
    tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817376k,mode=755)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
    tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1048560k)
    
    When we create another LXC container (on the same host node) from our debian-8.0-custom_amd64.tar.gz template (which was previously used for OpenVZ containers and works just fine for new LXC containers), the cgroup, mqueue, hugetlbfs mounts of the host system are getting exposed to the container:

    Code:
    ct186$ mount
    rpool/zfsdisks/subvol-186-disk-1 on / type zfs (rw,noatime,xattr,posixacl)
    none on /dev type tmpfs (rw,relatime,size=492k,mode=755)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
    proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
    proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
    sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
    sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
    sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
    fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
    lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
    devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
    devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
    devpts on /dev/tty1 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
    devpts on /dev/tty2 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
    tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
    cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
    cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event)
    cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb)
    cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids,release_agent=/run/cgmanager/agents/cgm-release-agent.pids)
    mqueue on /dev/mqueue type mqueue (rw,relatime)
    hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
    
    Can you please explain, why these mounts are getting exposed?

    The two containers were created via `pct create`:

    Code:
    $ pct create 185 local:vztmpl/debian-8.0-standard_8.4-1_amd64.tar.gz -hostname vtest4.example.com -cpulimit 2 -cpuunits 1024 -memory 4096 -swap 1024 -ostype debian -rootfs zfsvols:subvol-185-disk-1,acl=1 -net0 name=eth0,bridge=vmbr0,gw=XXX.XXX.XXX.XXX,ip=XXX.XXX.XXX.XXX/25 -storage zfsvols
    
    $ pct create 186 local:vztmpl/debian-8.0-custom_amd64.tar.gz -hostname vtest5.example.com -cpulimit 2 -cpuunits 1024 -memory 4096 -swap 1024 -ostype debian -rootfs zfsvols:subvol-186-disk-1,acl=1 -net0 name=eth0,bridge=vmbr0,gw=XXX.XXX.XXX.XXX,ip=XXX.XXX.XXX.XXX/25 -storage zfsvols
    
    While the debian-8.0-standard_8.4-1_amd64.tar.gz template was downloaded with `pveam`:

    Code:
    $ pveam update
    $ pveam available
    $ pveam download local debian-8.0-standard_8.4-1_amd64.tar.gz
    
     
  17. wbumiller

    wbumiller Proxmox Staff Member
    Staff Member

    Joined:
    Jun 23, 2015
    Messages:
    482
    Likes Received:
    47
    You can look at /etc/apparmor.d/abstractions/lxc/container-base to see what a container is by default allowed to do.
    mqueue, hugetlbfs, tmpfs, bind mounts of most paths and a few others are by default allowed. So these converted containers are mounting these file systems by themselves. Templates are often slightly modified, especially in their init process, and openvz differed quite a bit in that regard particularly. Some mounts might even be triggered by some services which might be running by default in one template but not in another.
     
  18. cpzengel

    cpzengel Member
    Proxmox VE Subscriber

    Joined:
    Nov 12, 2015
    Messages:
    61
    Likes Received:
    1
    have same here on lxc container with ispconfig
    mysqld does not start up
    i can rollback two days, everythinh is fine
    <<<--- running zfs

    apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns"
     

Share This Page