systemd 244.1 breaks networking in LXC

[moonman]

Hi everyone,

Just thought I would let everyone know that upgrading systemd to 244.1 will break networking if running in LXC container.
Found out the hard way and I'm probably one of the first ones since I'm running ArchLinux in LXC and systemd just got bumped up to 244.1.

Looking at systemd github there are quite a few networking changes between 244 and 244.1 https://github.com/systemd/systemd-stable/commits/v244-stable

Removing and adding network interface for the container in PVE interface for this container will get networking working again, until the container is restarted. I have not tried using something other than systemd-networkd.

EDIT: The solution:
Add lxc.mount.auto: sys to /etc/pve/lxc/<id>.conf

 

[oguz]

hi,

are you running privileged or unprivileged container?

 

[moonman]

Running unprivileged.

 

[latixia]

same issue here. my arch linux network was broken
privileged containers are not affected

 

[Qian Chen]

I'm using Arch as containers and problem solved by:
yay -U /var/cache/pacman/pkg/systemd-libs-244-1-x86\_64.pkg.tar.xz /var/cache/pacman/pkg/systemd-244-1-x86\_64.pkg.tar.xz

 

[moonman]

I'm using Arch as containers and problem solved by:
yay -U /var/cache/pacman/pkg/systemd-libs-244-1-x86\_64.pkg.tar.xz /var/cache/pacman/pkg/systemd-244-1-x86\_64.pkg.tar.xz

Well, everyone already knew that. yay is not the standard package manager. There is also https://archive.archlinux.org/ in case cache was cleared.
However, downgrading systemd packages is not a good long term solution.

 

[Qian Chen]

Well, everyone already knew that. There is also https://archive.archlinux.org/ in case cache was cleared.
However, downgrading systemd packages is not a good long term solution.

Of course this is not a long term solution. Yeah, I'm the last one to know it.

 

[moonman]

Yeah, I'm the last one to know it.

Fair enough. I guess not everyone then I added the solution to the OP.

 

[Qian Chen]

Fair enough. I guess not everyone then I added the solution to the OP.

You might want to remove the IgnorePkg after the next version of these packages are released.

 

[moonman]

You might want to remove the IgnorePkg after the next version of these packages are released.

Is there a fix coming in the next version?

 

[Qian Chen]

Is there a fix coming in the next version?

Not sure, but hopefully and likely next few versions, if not next one version. I have filed bug in both the systemd Github project and systemd package in Arch Linux website.

 

[moonman]

Tried more templates:

Fedora 30 does not work out of the box. Systemd v241-8.git9ef65cb.fc30
Fedora 31 works at first, stops after upgrading systemd to 243.5-1 it stops working. Working version is 243-4
Ubutnu 18.04 works. Systemd version 237

There must be a patch that's being backported to other versions and it is causing issues.

 

[Qian Chen]

@moonman I'm surprised no one else seems to be complaining this issue, not even anyone is talking about it. Are we the only two users of LXC on the whole internet? This in my opinion should be a major issue and everyone should jump and shout, I expected.

 

[moonman]

@moonman I'm surprised no one else seems to be complaining this issue, not even anyone is talking about it. Are we the only two users of LXC on the whole internet? This in my opinion should be a major issue and everyone should jump and shout, I expected.

My guess is that most LXC users run debian or some derivative that doesn't (yet) have this issue. Possibly CentOS which is very conservative with updates. The rest is not popular.

I'm sure someone is looking into this, it just takes time and it is the holiday season.
There is a workaround for now. Worse comes to worst I'll have to bisect commits between 244 - 244.1.

 

[Alwin]

@Elgs Qian Chen, thanks for opening the bug report upstream.
https://github.com/systemd/systemd/issues/14431

If anyone else experiences this, than please add your findings to the upstream bug report. Thanks.

EDIT: added lxc.mount.auto: sys to the CT config. It appears to work with the mixed mount mode of /sys.

 

[Qian Chen]

Thank you @Alwin, adding `lxc.mount.auto: sys` to /etc/pve/lxc/vmid.conf works!!

 

[moonman]

@Elgs Qian Chen, thanks for opening the bug report upstream.
https://github.com/systemd/systemd/issues/14431

If anyone else experiences this, than please add your findings to the upstream bug report. Thanks.

EDIT: added lxc.mount.auto: sys to the CT config. It appears to work with the mixed mount mode of /sys.

Thanks Alwin, it does work.
Are there any security concerns with this?

 

[Alwin]

Are there any security concerns with this?

Before that, sys was in RW mode.

 

[MadalinC]

Before that, sys was in RW mode.

Hi, I think you meant to say that /sas was ro before that and now it will be rw.

From what I can tell, you can continue to use /sys:ro and the functionality is working as expected without the risk of a writable /sys

 

[Alwin]

There are patches on our pve-devel mailing list, to streamline this again.
https://pve.proxmox.com/pipermail/pve-devel/2020-March/042424.html