Process hiding in LXC (hidepid proc mount option)

onlime

Renowned Member
Aug 9, 2013
76
14
73
Zurich, Switzerland
www.onlime.ch
How can I set proc (/proc) mount options for an LXC container on ProxmoxVE 4.2 (using ZFS storage)? I am talking about hidepid=2, which means: Users are only able too see their own processes (like with hidepid=1), but also the other process IDs are hidden for them in /proc.

On Proxmox 3.4 this was possible for OpenVZ containers simply by adding the following line to /etc/fstab inside the container:

Code:
proc /proc proc defaults,noexec,nosuid,nodev,hidepid=2 0 0

This won't work in a LXC container...

Code:
$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
 
besides, setting hidepid=2 on the ProxmoxVE host node has no effect on the LXC containers either.

Code:
hn$ mount -o remount,hidepid=2 /proc
hn$ mount | grep proc
proc on /proc type proc (rw,relatime,hidepid=2)

On the host node itself, this works just fine.
 
/proc inside a container is its own mount point.
In order to be able to remount as you suggested you'd need to allow this operation in the apparmor profile like this:
Code:
# /etc/apparmor.d/lxc/lxc-default-cgns-with-proc-remount
profile lxc-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # these are copied from lxc-container-default-cgns:
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,

  # This will allow remounting /proc, eg to change hidepid
  mount options=(rw, nosuid, nodev, noexec, remount, silent, relatime) -> /proc/,
}

The hidepid option doesn't need to be included there as it's not known by apparmor.

Reload via
Code:
 # apparmor_parser -r -W -T /etc/apparmor.d/lxc-containers

Set the profile for your container in /etc/pve/lxc/$VMID.conf:
Code:
(...)
lxc.aa_profile = lxc-default-cgns-with-proc-remount
 
Thanks @wbumiller for these detailed instructions. Couldn't get it running yet, though.

Reloading the new AppArmor profile with apparmor_parser did not report any errors.
Tried it with the following in /etc/pve/lxc/$VMID.conf (which is the correct syntax?):

Code:
lxc.aa_profile: lxc-default-cgns-with-proc-remount
# or ...
lxc.aa_profile = lxc-default-cgns-with-proc-remount

After restarting the LXC container, I tried:

Code:
$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
(...)

$ mount -o remount,hidepid=2 /proc
mount: cannot remount block device proc read-write, is write-protected

Code:
hn$ dmesg | grep apparmor
[337202.692601] audit: type=1400 audit(1472502879.621:202): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-default-cgns-with-proc-remount" name="/proc/" pid=9081 comm="mount" flags="rw, nosuid, nodev, noexec, remount"

Once I get this working, will I be able to auto-mount /proc via /etc/fstab as proposed?:

Code:
# <file system> <mount point>  <type>  <options>  <dump>  <pass>
proc     /proc     proc   defaults,noexec,nosuid,nodev,hidepid=2   0   0

Or do you suggest to remount it via init script?
 
Last edited:
Ah, your container isn't using the 'silent' and 'relatime' options required by the profile line I posted. (I just copied what I used in my test container). You'll have to match the `options=(flags)' portion of the AppArmor rule to your container (or add multiple rules for different possible combinations). Should be `options=(rw, nosuid, nodev, noexec, remount) according to the DENIED message output there and the mount flags seen above in your first post.
 
Great, thanks! Now it works.

Remounting worked with the following command:

Code:
$ mount -o remount,relatime,hidepid=2 /proc

I had to add the `relatime` flag as it was there before and as I put it into the AppArmor profile as required option. Don't quite get it, why I need to specify it explicitly on the remount command.
Here's my working solution...

hn:/etc/apparmor.d/lxc/lxc-default-cgns-with-proc-remount

Code:
# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,

  # This will allow remounting /proc, eg to change hidepid
  mount options=(rw, nosuid, nodev, noexec, remount, relatime) -> /proc/,
}

hn:/etc/pve/lxc/$VMID.conf

Code:
(...)
lxc.aa_profile: lxc-container-default-cgns-with-proc-remount

reload AppArmor profiles:

Code:
hn$ apparmor_parser -r -W -T /etc/apparmor.d/lxc-containers

ct:/etc/fstab

Code:
# <file system> <mount point>  <type>  <options>  <dump>  <pass>
proc  /proc  proc  defaults,noexec,nosuid,nodev,relatime,hidepid=2 0  0

restart LXC container:

Code:
hn$ pct stop $VMID && pct start $VMID

/proc will then be correctly mounted with the `hidepid=2` option. No remounting required!

Code:
ct$ mount | grep 'proc on'
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime,hidepid=2)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime,hidepid=2)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime,hidepid=2)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime,hidepid=2)
 
Last edited:
I am still getting the following DENIED apparmor messages upon container start:

Code:
[376680.591775] audit: type=1400 audit(1472542357.517:395): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/" pid=8574 comm="mount" flags="rw, remount"
[376680.594481] audit: type=1400 audit(1472542357.521:396): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/" pid=8575 comm="mount" flags="rw, remount, noatime"
[376680.797361] audit: type=1400 audit(1472542357.725:397): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/run/" pid=8816 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
[376680.812879] audit: type=1400 audit(1472542357.741:398): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/run/lock/" pid=8833 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[376680.833557] audit: type=1400 audit(1472542357.761:399): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/sys/" pid=8854 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
[376680.986368] audit: type=1400 audit(1472542357.913:400): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/run/shm/" pid=9018 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[376680.998105] audit: type=1400 audit(1472542357.925:401): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns-with-proc-remount" name="/dev/pts/" pid=9029 comm="mount" flags="rw, nosuid, noexec, remount, relatime"

This also happens on an fresh LXC container with the default lxc-container-default-cgns AppArmor profile:

Code:
[376624.608565] audit: type=1400 audit(1472542301.537:387): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=5212 comm="mount" flags="rw, remount"
[376624.610775] audit: type=1400 audit(1472542301.537:388): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=5213 comm="mount" flags="rw, remount"
[376624.845170] audit: type=1400 audit(1472542301.773:389): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/" pid=5481 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
[376624.859926] audit: type=1400 audit(1472542301.785:390): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/lock/" pid=5498 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[376624.870490] audit: type=1400 audit(1472542301.797:391): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/proc/" pid=5508 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[376624.882287] audit: type=1400 audit(1472542301.809:392): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/sys/" pid=5519 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
[376625.033363] audit: type=1400 audit(1472542301.961:393): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/shm/" pid=5683 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[376625.045081] audit: type=1400 audit(1472542301.973:394): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/dev/pts/" pid=5694 comm="mount" flags="rw, nosuid, noexec, remount, relatime"

Should I report this as a new issue?
 
What kind of container is it? And is it converted from openvz or fresh?
 
It was a container which was converted from OpenVZ. Basically I went this direct migration path (migration via `vzdump` and `pct restore` definitely is no option for us and I am going to propose the following as alternative migration path):

  1. Stopped all containers (actually replicated them via ZREP to secondary host node and running them there, still on ProxmoxVE 3.4/OpenVZ)
  2. Upgraded primary host node from ProxmoxVE 3.4 to 4.2
  3. Create CT with same VMID via ProxmoxVE WebUI, choosing any template for the new LXC container
  4. moving the existing ZFS volume in place:
Code:
$ VMID=184
$ zfs destroy rpool/zfsdisks/subvol-$VMID-disk-1
$ zfs rename rpool/ROOT/pve-$VMID rpool/zfsdisks/subvol-$VMID-disk-1
$ zfs set mountpoint=/rpool/zfsdisks/subvol-$VMID-disk-1 rpool/zfsdisks/subvol-$VMID-disk-1

$ pct start $VMID

This worked fine. When I check your http://pve.proxmox.com/wiki/Convert_OpenVZ_to_LXC#Step_by_step_conversion instructions it looks like nothing needs to be changed inside the container filesystem.

I have also just set up a fresh LXC container via the ProxmoxVE WebUI and with a fresh basic system image:

Code:
$ pveam update
$ pveam available
$ pveam download local debian-8.0-standard_8.4-1_amd64.tar.gz

Once I start this new container, I'll get the same apparmor DENIED messages:

Code:
$ pct start $VMID
$ dmesg | grep apparmor
[380948.425404] audit: type=1400 audit(1472546625.352:424): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/sys/fs/pstore/" pid=26644 comm="mount" fstype="pstore" srcname="pstore"
[380948.425466] audit: type=1400 audit(1472546625.352:425): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/sys/fs/pstore/" pid=26644 comm="mount" fstype="pstore" srcname="pstore" flags="ro"
[380948.557403] audit: type=1400 audit(1472546625.484:426): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=26753 comm="mount" flags="rw, remount, silent"
[380948.559915] audit: type=1400 audit(1472546625.484:427): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=26754 comm="mount" flags="rw, remount, noatime"
[380948.620682] audit: type=1400 audit(1472546625.548:428): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/" pid=26809 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
[380948.631870] audit: type=1400 audit(1472546625.556:429): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/lock/" pid=26820 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[380948.639589] audit: type=1400 audit(1472546625.564:430): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/proc/" pid=26827 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[380948.649852] audit: type=1400 audit(1472546625.576:431): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/sys/" pid=26835 comm="mount" flags="ro, nosuid, nodev, noexec, remount, relatime"
[380948.719792] audit: type=1400 audit(1472546625.644:432): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/run/shm/" pid=26901 comm="mount" flags="rw, nosuid, nodev, noexec, remount, relatime"
[380948.729537] audit: type=1400 audit(1472546625.656:433): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default-cgns" name="/dev/pts/" pid=26909 comm="mount" flags="rw, nosuid, noexec, remount, relatime"
 
Ah yes, they're expected in some guest OS types. Some file systems get mounted by lxc before starting up the guest system, which is then not allowed to replace the existing mount points. Not all guest systems produce the same warnings, though. There are fewer such messages in container-aware guests (specifically ones using a new-enough systemd).
 
Could you please describe what sysrq-trigger and relatime for proc inside a container should do? Or is this still some leftovers from OpenVZ?
 
Could you please describe what sysrq-trigger and relatime for proc inside a container should do? Or is this still some leftovers from OpenVZ?

Good question! I was also quite confused about these new defaults. I am comparing a plain Debian Jessie OpenVZ container on ProxmoxVE 3.4:

Code:
$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

vs. a plain Debian Jessie LXC container on ProxmoxVE 4.2:

Code:
$ mount | grep proc
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)

`relatime` was already there in OpenVZ, even on the `/proc` mount of the host node. AFAIK, this was always the default. Don't ask me if that makes sense on a proc filesystem...

Concerning the `/proc/sysrq-trigger` and all the other proc mounts, I would really appreciate if someone could shed some light.
 
/proc/sys contains system settings containers should not have write access to, so lxc mounts it as read-only
/proc/sys/net contains the network part of /proc/sys, which is namespaced and thus can (and should) be read-write (in order to change interface settings such as forwarding, or disabling/enabling ipv6, autoconfiguration, rp-filtering, ...)
/proc/sysrq-trigger can do lots of nasty stuff[1] and is therefore double protected via a read-only mount and an AppArmor rule.

I suppose OpenVZ dealt with that via kernel changes, but I haven't checked.

[1] https://www.kernel.org/doc/Documentation/sysrq.txt
 
Thank you Wolfgang. I was not aware of the fact that /proc is actually mounted without "shielding" important stuff like sysrq automatically. I cannot think of a use case in where it makes sense to have sysrq inside a container, so it should be hidden by the kernel itself.
 
Except the kernel doesn't have a concept of containers as such. It just provides building blocks such as namespaces and cgroups, which can also be used independently and for different purposes. I frequently use network namespaces via `ip netns` for tunnels/vpns or stuff I want to isolate only from the network, or mount namespaces (via `unshare`) to hide certain mounts from other unprivileged prying eyes.
So to the kernel, you're just mounting yet another /proc.
 
Last edited:
Thanks for the explanation @wbumiller
The following might be a bit off-topic, but I would finally like to understand the default mounts inside an LXC container...

We now run into another weird issue. If we create a new LXC container from the debian-8.0-standard_8.4-1_amd64.tar.gz template, we see the following mounts:

Code:
ct185$ mount
rpool/zfsdisks/subvol-185-disk-1 on / type zfs (rw,noatime,xattr,posixacl)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty1 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty2 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817376k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=1048560k)

When we create another LXC container (on the same host node) from our debian-8.0-custom_amd64.tar.gz template (which was previously used for OpenVZ containers and works just fine for new LXC containers), the cgroup, mqueue, hugetlbfs mounts of the host system are getting exposed to the container:

Code:
ct186$ mount
rpool/zfsdisks/subvol-186-disk-1 on / type zfs (rw,noatime,xattr,posixacl)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys/net type proc (rw,nosuid,nodev,noexec,relatime)
proc on /proc/sys type proc (ro,nosuid,nodev,noexec,relatime)
proc on /proc/sysrq-trigger type proc (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (ro,nosuid,nodev,noexec,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,relatime)
sysfs on /sys/devices/virtual/net type sysfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
lxcfs on /proc/cpuinfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/diskstats type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/meminfo type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/stat type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/swaps type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
lxcfs on /proc/uptime type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty1 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
devpts on /dev/tty2 type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset,clone_children)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids,release_agent=/run/cgmanager/agents/cgm-release-agent.pids)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)

Can you please explain, why these mounts are getting exposed?

The two containers were created via `pct create`:

Code:
$ pct create 185 local:vztmpl/debian-8.0-standard_8.4-1_amd64.tar.gz -hostname vtest4.example.com -cpulimit 2 -cpuunits 1024 -memory 4096 -swap 1024 -ostype debian -rootfs zfsvols:subvol-185-disk-1,acl=1 -net0 name=eth0,bridge=vmbr0,gw=XXX.XXX.XXX.XXX,ip=XXX.XXX.XXX.XXX/25 -storage zfsvols

$ pct create 186 local:vztmpl/debian-8.0-custom_amd64.tar.gz -hostname vtest5.example.com -cpulimit 2 -cpuunits 1024 -memory 4096 -swap 1024 -ostype debian -rootfs zfsvols:subvol-186-disk-1,acl=1 -net0 name=eth0,bridge=vmbr0,gw=XXX.XXX.XXX.XXX,ip=XXX.XXX.XXX.XXX/25 -storage zfsvols

While the debian-8.0-standard_8.4-1_amd64.tar.gz template was downloaded with `pveam`:

Code:
$ pveam update
$ pveam available
$ pveam download local debian-8.0-standard_8.4-1_amd64.tar.gz
 
You can look at /etc/apparmor.d/abstractions/lxc/container-base to see what a container is by default allowed to do.
mqueue, hugetlbfs, tmpfs, bind mounts of most paths and a few others are by default allowed. So these converted containers are mounting these file systems by themselves. Templates are often slightly modified, especially in their init process, and openvz differed quite a bit in that regard particularly. Some mounts might even be triggered by some services which might be running by default in one template but not in another.
 
have same here on lxc container with ispconfig
mysqld does not start up
i can rollback two days, everythinh is fine
<<<--- running zfs

apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns"
 
The apparmor profile solution worked fine until Proxmox 5.0 released LXC 2.1. Now, it no longer works, /proc is no longer mounted with hidepid=2 mount option inside the container.

That's how I tried:

Code:
# /etc/apparmor.d/lxc/lxc-default-cgns-with-proc-remount

# Do not load this file.  Rather, load /etc/apparmor.d/lxc-containers, which
# will source all profiles under /etc/apparmor.d/lxc

profile lxc-container-default-cgns-with-proc-remount flags=(attach_disconnected,mediate_deleted) {
  #include <abstractions/lxc/container-base>

  # the container may never be allowed to mount devpts.  If it does, it
  # will remount the host's devpts.  We could allow it to do it with
  # the newinstance option (but, right now, we don't).
  deny mount fstype=devpts,
  mount fstype=cgroup -> /sys/fs/cgroup/**,

  # This will allow remounting /proc, e.g. to add hidepid=2 mount option
  # The hidepid option doesn't need to be included here as it's not known by AppArmor.
  mount options=(rw,nosuid,nodev,noexec,remount,relatime) -> /proc/,
}

loading it:

Code:
$ apparmor_parser -r -W -T /etc/apparmor.d/lxc-containers

adding to the container's /etc/fstab (inside the container):

Code:
proc  /proc  proc  defaults,noexec,nosuid,nodev,relatime,hidepid=2 0  0

and to the container config, using the new lxc.apparmor.profile key:

Code:
$ pct config $VMID | grep lxc
lxc.apparmor.profile: lxc-container-default-cgns-with-proc-remount

then restarting the CT:

Code:
$ pct shutdown $VMID
$ pct start $VMID

inside the container:

Code:
$ mount | grep '/proc '
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

Any idea how to fix this in ProxmoxVE 5.0 / LXC 2.1 ?
 
Last edited:
BUMP

I still cannot get `hidepid=2` proc mount option running on latest Proxmox VE 5.1 (4.13.13-6-pve kernel). I tried it again using lxc.apparmor.profile as exactly described in my previous post from Sep 29, 2017.

Won't work and I am not getting any apparmor error messages in syslog, checked both PVE host and LXC container logs.

I'll definitely invite you for a beer if you figure this out! I may even pay you the flight to get here.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!