LXCFS not mounting (on 4.2 pvetest) - breaks startup/systemd for some containers

[f948lan]

Following up with a new thread on my experience as requested...

Symptoms (for anyone reading this in future):
- /proc/cpuinfo, /proc/uptime and other lxcfs provided files are empty in containers
- containers using systemd > v215 fail to startup correctly (no services start, thus no getty login, no ifconfig)
- but no error in Proxmox GUI.
Starting in foreground (lxc-start -F -lDEBUG --name <ct_id>) shows some startup errors but with no useful clue to the cause.

After your feedback on my previous comments, that a new Jessie install was fine for you, I've been working through what I did again to document the steps trying to reproduce - and believe I've found the problem step (which is basically my fault - no surprises there - but I'm sure I won't be the last person to be caught by this, so I still suggest a behaviour change below...)

I was basically following the instructions here:
https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie

1) New Debian 8 install from Network Installer/Minimal boot CD iso
GPT, UEFI boot, separate ext2 /boot partition, ext4 /
2) Select Base packages only for install
3) Basic config as per above HowTo
3) Add the Proxmox pvetest repo
4) apt update, apt sys-upgrade

Now this is where I think I broke it...
Boot drive is an SSD and I wanted to give F2FS a trial too. As the Jessie installer doesn't natively support F2FS, I installed to a small ext4 partition.
So at this point made new F2FS partition and copied the ext4 root over to it (and updated initramfs, grub and fstab appropriately, and finally made the old ext4 partition swap).
I only intended to copy the root filesystem to F2FS, but it looks like I copied the mounted lxcfs data too. Thus after the reboot to my new root partition, lxcfs did not mount, as the default behaviour is not to mount if the mountpoint is not empty. This behaviour comes from fuse rather than lxcfs itself, but looking at lxcfs there is no way to pass the nonempty option to fuse without changing the source. Frankly for something as essential as lxcfs in this scenario, the nonempty flag should be default, the risks of not using it are greater.

However, prevention is better than cure, so my proposed improvement:
Add a sanity check to the lxcfs startup, which cleans /var/lib/lxcfs before mounting (nothing important should ever be in that location).

(At the very least, some GUI warning if lxcfs is not mounted, or check for it when starting a container should be implemented to make it easier for users suffering similar issues to resolve the problem.)

Could you open a new thread for this with the exact steps you did to run into this issue? A clean minimal install of jessie and following the HowTo at https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie works for me (lxcfs is correctly installed and started).

Please include as much information as possible (which jessie installation image, which selections in the installer, exact steps after the jessie installation, any error messages, ..).

Further to my experience below, I've now found the issue/solution for systemd on Ubuntu containers with the latest pvetest on a clean jessie install.

On top of my inital exerience below, I also noticed that /proc/cpuinfo, /proc/uptime and other (lxcfs delivered) files were empty in all containers, which some packages understandably did not like (including newer systemd it seems).

I'm not sure if it affect users who are upgrading (maybe not as I don't see anyone else reporting this issue?), but certainly for a clean jessie+pvetest install, lxcfs does not start on the node. This is a result of /var/lib/lxcfs being populated during install with empty files, and lxcfs refusing to start on an non-empty directory. The workaround in is to delete the contents of /var/lib/lxcfs and restart the node (or manually start lxcfs). After that everything is working much better!

Please can you fix this in the next update to pvetest?

Also, on a different note, I saw others asking about supporting containers from images.linuxcontainer.org more easilly. In fact it seems the actual setup process is quite happy detecting the compression and working from .tar.xz images. It's just the check that's done during upload needs tweaking to accept .tar.xz as well as .tar.gz - renaming a tar.zx to tar.gz to trick the upload test will install fine on pvetest - proven with a nighly CentOS container. Any chance of adding this small change for the next release?

 

[fabian]

Thanks for the detailled report!

However, prevention is better than cure, so my proposed improvement:
Add a sanity check to the lxcfs startup, which cleans /var/lib/lxcfs before mounting (nothing important should ever be in that location).

I will talk with upstream to see whether we can fix it one way or another (see below).

(At the very least, some GUI warning if lxcfs is not mounted, or check for it when starting a container should be implemented to make it easier for users suffering similar issues to resolve the problem.)

Availability of lxcfs is checked upon container start up - but unfortunately only on the cgroup / controller level, and not for the FUSE mountpoint specifically. This would not be a problem per se, but the FUSE mountpoint is established last in the lxcfs startup sequence, and the controllers are not removed again if the mount fails. Which leads to the situation you experience - lxcfs startup fails because of the failing FUSE mount, but the lxcfs controllers exist and allow containers to start with a broken lxcfs (at least from an lxc point of view, if you require stuff provided by lxcfs in order to boot, that will obviously not work).