PSA -- Do not upgrade to systemd 247

[inxsible]

I have multiple archlinux LXC containers and I updated one of them and I no longer had networking after a reboot of that container.

I fired up a new container for debugging and updated the Archlinux packages and systemd was upgraded to 247. During investigation, I found that a bunch of systemd services were failing.

[checkarch: ~]── - systemctl --version

systemd 247 (247.2-1-arch)

+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid

[checkarch: ~]── - status systemd-networkd
* systemd-networkd.service - Network Service
     Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2020-12-18 16:02:11 UTC; 3h 39min ago
TriggeredBy: * systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
    Process: 347 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=226/NAMESPACE)
   Main PID: 347 (code=exited, status=226/NAMESPACE)

Dec 18 16:02:11 checkarch systemd[1]: systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE
Dec 18 16:02:11 checkarch systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Dec 18 16:02:11 checkarch systemd[1]: Failed to start Network Service.
Dec 18 16:02:11 checkarch systemd[1]: systemd-networkd.service: Scheduled restart job, restart counter is at 5.
Dec 18 16:02:11 checkarch systemd[1]: Stopped Network Service.
Dec 18 16:02:11 checkarch systemd[1]: systemd-networkd.service: Start request repeated too quickly.
Dec 18 16:02:11 checkarch systemd[1]: systemd-networkd.service: Failed with result 'exit-code'.
Dec 18 16:02:11 checkarch systemd[1]: Failed to start Network Service.

No amount of prodding would bring up the interfaces. Eventually I found that every systemd service that fails needs to be overridden to allow mounting /proc along with /sys

sudo systemctl edit systemd-networkd

OR

sudo vim /etc/systemd/system/systemd-networkd.service.d/override.conf

Add the following into the override.conf

[Service]
BindReadOnlyPaths= /sys /proc

Save and then restart the service and it should start up. Trouble is that there a bunch of services which fail and you might have to override each one of them depending on your usage.

I think this issue is because of the difference between the host(proxmox) and the guest(archlinux). I have other Archlinux desktop/laptop which have not had any problems during the upgrades.

So it would be best to avoid upgrading systemd on your containers-- especially if the container distro is rolling release/bleeding edge.


Hope that helps !

 

[fabian]

https://github.com/systemd/systemd/issues/17866

 

[inxsible]

https://github.com/systemd/systemd/issues/17866

Thanks, but it seems after it being initially dismissed by poettering as an LXC or LXC/apparmor issue, the other devs are trying to fix it but have not yet found an optimum solution.

I guess I will wait to upgrade my other Archlinux containers until there is some resolution.

 

[fabian]

it's not the first time something like that has come up, I am sure either a fix on the systemd side or a workaround on the LXC side will eventually materialize

 

[nhand42]

after it being initially dismissed by poettering

Recurring theme. Thanks for the workaround.

 

[wbumiller]

An alternative to the systemd overrides would be to allow systemd to do its thing (but I only recommend this for unprivileged containers) via

# Append to /etc/pve/lxc/<arch ct ids>.conf
lxc.apparmor.raw: mount fstype=proc options=(nosuid,nodev,noexec,rw) -> /run/systemd/unit-root/proc/,

(Note the trailing comma!)

This kind of stuff is probably going to happen with every other systemd release.
Systemd will keep using a lot of combinations of mount options and the only way to allow them all is to allow either too much (security concerns...) or to write out all combinations out explicitly... (lxc currently generates a large amount of rules on the fly for bind mounts for instance, also in response to systemd, but full proc mounts like this haven't been covered yet).
By default some mount rules are restricted to certain paths, because access to some parts of those file systems are also restricted, which can only be done by-path, so mounting it some place else requires special care.
`/proc/sys/kernel` is for instance denied by the apparmor rules, so in this case, that's something to be aware of. However, this is only an issue with privileged containers.

So basically this means arch containers will require some extra permissions every so often for now, which is why I do recommend sticking to unprivileged containers and therefore "moving" some security measures off of apparmor and to the kernel's user namespace implementation.
There isn't much magic we can do here as an alternative.

 

[inxsible]

Thanks @wbumiller, your explanation makes sense.

It is definitely better to simply make one change in the container's config file rather than having to override every systemd service that is failing. I only have unprivileged containers for every service that I run and have not had the need to create privileged containers yet. I will try this out later in one of my containers and see how it goes.

 

[inxsible]

I removed the systemd unit overrides that I had in 1 of my Archlinux containers and replaced it with the lxc.apparmor.raw configuration that @wbumiller suggested and I can confirm that the systemd units do NOT fail.

So it's still a workaround, but one that is isolated only to the Container's config file rather than in multiple individual systemd files.

Thanks again.

 

[inxsible]

An update:
Last evening I updated my Archlinux containers and after rebooting a couple of them, I had no networking again. This was surprising because I already had the apparmor configuration in the config files for those containers as suggested by @wbumiller a few posts above.

Did some more research and then I enabled the Nesting feature and voila, the networking was up again. Then for testing, I removed the lxc.apparmor.raw configuration from the config file and rebooted to check if the network was still up and it was.

So it seems that the latest update (I updated proxmox as well as the container's systemd) seems to fail with the configuration that I had put back in Jan. Now the only thing it needs is to set the Nesting feature for that container under Options-->Features-->Edit-->Nesting


I'll continue to monitor this container before I go ahead and reboot the other containers. If everything works for the next few days, I will go ahead and enable Nesting on each and remove the apparmor mount configuration from the config files. As previously mentioned, all my containers are unprivileged so I am ok enabling Nesting, but if you use privileged containers, then you might want to hold off.

 

[fabian]

An update:
Last evening I updated my Archlinux containers and after rebooting a couple of them, I had no networking again. This was surprising because I already had the apparmor configuration in the config files for those containers as suggested by @wbumiller a few posts above.

Did some more research and then I enabled the Nesting feature and voila, the networking was up again. Then for testing, I removed the lxc.apparmor.raw configuration from the config file and rebooted to check if the network was still up and it was.

So it seems that the latest update (I updated proxmox as well as the container's systemd) seems to fail with the configuration that I had put back in Jan. Now the only thing it needs is to set the Nesting feature for that container under Options-->Features-->Edit-->Nesting


I'll continue to monitor this container before I go ahead and reboot the other containers. If everything works for the next few days, I will go ahead and enable Nesting on each and remove the apparmor mount configuration from the config files. As previously mentioned, all my containers are unprivileged so I am ok enabling Nesting, but if you use privileged containers, then you might want to hold off.

also see our devel list for ongoing work to improve the usability there: https://lists.proxmox.com/pipermail/pve-devel/2021-August/049615.html