Interesting LXC & systemd findings

sigxcpu

Well-Known Member
May 4, 2012
433
11
58
Bucharest, Romania
I have my own ubuntu-15.04 installed using debootstrap. It looked being systemd-based.

This template worked since Proxmox 4 Beta 1 without issues. Today I've upgraded to -50 and also took the chance to reboot (was still running kernel 4.2.1).

All the LXC containers based on this template failed to start. I mean the container itself started but "pct enter" shown 4 processes running as it looks like most of the people hit on non-Proxmox approved templates.

I've fixed the issue by installing upstart-sysv (that replaced systemd-sysv).

Now, what bothers me, although the general approach is to not have a systemd-based template for LXC, mine worked without a hitch. It looks like something recently updated broke this functionality.

Here's the output for lxc-start: https://gist.github.com/sigxcpu76/ff0d1ea48180339baae4
Here is the process list inside the container:

Code:
# ps uaxwww
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.3  34052  2040 ?        Ss   00:54   0:00 /sbin/init
root        79  0.0  0.4  18248  2268 ?        S    00:55   0:00 /bin/bash
root        84  0.0  0.2  15600  1328 ?        R+   00:55   0:00 ps uaxwww

Any ideas?
 
Further search brought me to something related to lxcfs. Here's the output from a working server, inside the container:

Code:
# ls -l /sys/fs/cgroup/systemd/lxc/
total 0
drwxr-xr-x 2 root root 0 Oct 20 01:47 100

Here's the output from a non-working server, inside the container:

Code:
# ls -l /sys/fs/cgroup/systemd/
ls: cannot access /sys/fs/cgroup/systemd/lxc: No such file or directory
total 0
?????????? ? ? ? ?            ? lxc

Both containers are based on my ubuntu 15.04 template, systemd 219.

Both hypervisors are running latest pvetest versions: kernel 4.2.2-1-pve, lxc-pve 1.1.3-1 and lxcfs 0.9-pve2. I really don't understand what the difference is.
 
It is reproductible. The server that doesn't work, never works. The one that does, does it all the time :)

I can provide the systemd-based 15.04 template to you, if you want (or a simple apt-get install systemd-sysv will do it in a standard sysv container).

What is the current view of Proxmox team on systemd-based containers? Are they officially supported or not?

Maybe someone in the team that is more LXC-savvy can ask me to do some specific debugging on the broken host.

All I did was to strace systemd --system inside the container and that shown me the broken /sys/.../lxc folder. Now we need to find out why lxcfs has that entry broken.
 
Last edited:
It is reproductible. The server that doesn't work, never works. The one that does, does it all the time :)

Sure. I meant reproducible for us.

I can provide the systemd-based 15.04 template to you, if you want.

better provide instruction howto create such template.

What is the current view of Proxmox team on systemd-based containers? Are they officially supported or not?

AFAIK our ubuntu 15.04 template is also systemd based. We try to support it, but systemd people still make changes that
do not work with current lxc (see sysetmd version 226).

All I did was to strace systemd --system inside the container and that shown me the broken /sys/.../lxc folder. Now we need to find out why lxcfs has that entry broken.

Does it help if you reboot that node (to make sure all service are running correctly)?
 
All of these happened after a reboot. Actually it started to not boot these containers after the reboot (prev kernel was 4.2.1 from a beta PVE4).

Anyway, this doesn't look like a systemd issue. We cannot blame systemd because it fails to access a broken folder. Here's the "strace systemd --system" output.

Code:
....
newfstatat(AT_FDCWD, "/sys/fs/cgroup/systemd", {st_dev=makedev(0, 67), st_ino=16, st_mode=S_IFDIR|0755, st_nlink=2, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=0, st_size=0, st_atime=2015/10/20-18:47:49, st_mtime=2015/10/20-18:47:49, st_ctime=2015/10/20-18:47:49}, AT_SYMLINK_NOFOLLOW) = 0
stat("/sys/fs/cgroup/systemd/lxc", 0x7ffd717e3810) = -1 ENOENT (No such file or directory)
mkdir("/sys", 0755)                     = -1 EEXIST (File exists)
mkdir("/sys/fs", 0755)                  = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup", 0755)           = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/systemd", 0755)   = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/systemd/lxc", 0755) = -1 ENOENT (No such file or directory)
writev(2, [{"\33[1;31m", 7}, {"Failed to create root cgroup hie"..., 65}, {"\33[0m", 4}, {"\n", 1}], 4Failed to create root cgroup hierarchy: No such file or directory
) = 77
lstat("/sys/fs/cgroup/systemd/lxc/103", 0x7ffd717e3830) = -1 ENOENT (No such file or directory)
epoll_ctl(4, EPOLL_CTL_DEL, 5, NULL)    = 0
close(5)                                = 0
close(4)                                = 0
writev(2, [{"\33[1;31m", 7}, {"Failed to allocate manager objec"..., 60}, {"\33[0m", 4}, {"\n", 1}], 4Failed to allocate manager object: No such file or directory
) = 72
close(3)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++

And as I've pasted above:

Code:
# ls -l /sys/fs/cgroup/systemd
ls: cannot access /sys/fs/cgroup/systemd/lxc: No such file or directory
total 0
?????????? ? ? ? ?            ? lxc

So this looks more like a lxcfs issue, that folder being mounted by it:

Code:
# mount | grep systemd
lxcfs on /sys/fs/cgroup/systemd type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)



systemd doesn't even have the chance to start, because that folder is corrupted by lxcfs.

I've switched the container to upstart-sysv and, as expected, all the service boot, but here's the look of that folder again:

Code:
root@testlxc:~# ls -l /sys/fs/cgroup/systemd/
ls: cannot access /sys/fs/cgroup/systemd/lxc: No such file or directory
total 0
?????????? ? ? ? ?            ? lxc

So lxcfs presents corrupted systemd folder structure even with a working container.

If somebody can guide me where to look for that folder mapping can be a start to dig further. I've looked in lxc mount hooks but those look to address something else.
 
check if this bug is part of the issue: https://bugzilla.proxmox.com/show_bug.cgi?id=773

and the the lxc conf files are what you expect.

Nope, it is not a .conf issue. The .conf file (that is transformed to a standard lxc config file on each startup) does not have information on the pseudo-mounts done by the LXC subsystem using lxcfs. Those mounts are done by the LXC subsystem and system-level included common configs (e.g. ubuntu.conf).
 
Because the folder is present as an entry but its rights and other stat data is invalid (represented by "?"s). The ls is done in the parent directory that shows an invalid "lxc" entry.

Ah, I see. I would still help if we can reproduce it here...
 
Did you check that folder inside a container on different machines? I told, my software versions are the same on both matchines (apt-get update && apt-get dist-upgrade using testing repository). Both machines rebooted, having the latest kernel. They behave differently.

So there is a chance that one the containers on of your test machines has that folder corrupted (although not affected if not using systemd).
 
Solved for the non-working machine. Took the lxcfs sources (master which is 0.10 with some patches) from https://github.com/lxc/lxcfs. Compiled and replaced /usr/bin/lxcfs only (no make install).

It works everytime. The systemd folder is not corrupted anymore and, as expected, systemd fully starts with all the services.
 
Last edited:
It works everytime. The systemd folder is not corrupted anymore and, as expected, systemd fully starts with all the services.

Thats for tracking that down - I already updated our git repositories, and will upload new version asap.
 
Does this fix the issue with tempfs not working inside the lxc containers? For example, no service (such as NTP, httpd, ect) works without making .conf file changes inside the container's systemd settings for that service.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!