LXC: cannot mount /run inside guest after PVE 8 to 9 upgrade (was: loopback interface not coming up)

[Le_Coyote]

Hello,

Long time user, first time posting, because everything had been working so well so far!

I have a few CTs running Linux Devuan 6 Excalibur. The network configuration is nothing out of the ordinary, and pretty much a copy/paste save for the MAC and IP address obviously. They are set up using veth on the bridge with a publicly reachable IP.

The issue is that two of these do not bring up the loopback interface at boot. From within the container, simply issuing an "ifconfig lo up" is enough to cure the problem, but I cannot figure out WHY it's not coming up on its own in these two containers, even when /etc/network/interfaces has the very same "auto lo …" stanza.

Has anyone encountered this behaviour before? I have been unable to find anything similar on the forum so far.
How would I go about investigating this?

Many thanks!
EDIT: for context, containers are running on PVE 9.1.16

 

[Le_Coyote]

Hello

I have confirmed that this problem emerges after upgrading a node from PVE 8 to 9. In other words, the guest was working very well in PVE8, but in 9, network breaks down. It does not happen for all the guests, but I have not been able to figure out why some work and others won't. And I have reproduced the issue with 3 different guests on two different physical nodes.
One of the nodes was even brought up without any interface at all after the upgrade. By deleting and re-creating the network interface from the PVE GUI, eth0 came up again, but not lo.

Could it be some edge case scenario with devuan? Are there node-side scripts that I should look for in order to investigate further?
Thanks
EDIT: additional info

 

[Le_Coyote]

Hello again.

After some digging, it looks like the network issue is a side effect of not being able to mount /run. From the node's journal:
kernel: audit: type=1400 audit(1779270973.875:168): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxc-100053_</var/lib/lxc>" name="/run/" pid=105111 comm="mount" flags="rw, move"

On the other nodes, I have no such errors.
What could cause this?
Changing title for clarity

 

[sterzy]

Can you post the configuration of the container? Simply run pct config <CT-ID> (replace <CT-ID> with the container's ID of course), then post the result here, ideally between [code][/code] tags. Thank you!

 

[Le_Coyote]

Can you post the configuration of the container? Simply run pct config <CT-ID> (replace <CT-ID> with the container's ID of course), then post the result here, ideally between [code][/code] tags. Thank you!

Of course:

arch: amd64
cores: 1
hostname: ns0.[redacted].org
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=[redacted],gw6=[...],hwaddr=[...],ip=[...]/32,ip6=[...]/64,type=veth
onboot: 1
ostype: devuan
rootfs: vmdata:subvol-100053-disk-0,size=8G
swap: 512
unprivileged: 1

What strikes me as odd regarding apparmor is that, in /var/lib/lxc/100053/apparmor/lxc-100053_\<-var-lib-lxc\>, there is

# allow tmpfs mounts everywhere
mount fstype=tmpfs,

, and no other stanza anywhere seems to prohibit mounting of tmpfs on /run. The apparmor rules are exactly the same as on the guests where it works.

Thanks for taking a look

 

[Le_Coyote]

A quick update.
First of all, even though the guests are not running systemd, nesting=1 is enough to solve the issue. I am wondering why.
Second, I was under the impression that some guests worked, but it was only because it was their first boot, and the non-tmpfs /run was clean. Subsequent boots of guests also display the same unexplained behaviour, ie. apparmor forbids mounting of /run and a non-clean /run means no/buggy network.

In other words, nesting=1 looks required by devuan, but I would love to hear why, as well as why the "allow tmpfs" from the apparmor profile is not enough.

Thanks!

 

[sterzy]

Thanks for the updates! I'll see if I can give you a more concrete answer once I found out a little about what's going on here.