LXC: cannot mount /run inside guest after PVE 8 to 9 upgrade (was: loopback interface not coming up)

L

Le_Coyote

New Member
May 19, 2026
10
1
3
Hello,

Long time user, first time posting, because everything had been working so well so far!

I have a few CTs running Linux Devuan 6 Excalibur. The network configuration is nothing out of the ordinary, and pretty much a copy/paste save for the MAC and IP address obviously. They are set up using veth on the bridge with a publicly reachable IP.

The issue is that two of these do not bring up the loopback interface at boot. From within the container, simply issuing an "ifconfig lo up" is enough to cure the problem, but I cannot figure out WHY it's not coming up on its own in these two containers, even when /etc/network/interfaces has the very same "auto lo …" stanza.

Has anyone encountered this behaviour before? I have been unable to find anything similar on the forum so far.
How would I go about investigating this?

Many thanks!
EDIT: for context, containers are running on PVE 9.1.16
 
Last edited:
Hello

I have confirmed that this problem emerges after upgrading a node from PVE 8 to 9. In other words, the guest was working very well in PVE8, but in 9, network breaks down. It does not happen for all the guests, but I have not been able to figure out why some work and others won't. And I have reproduced the issue with 3 different guests on two different physical nodes.
One of the nodes was even brought up without any interface at all after the upgrade. By deleting and re-creating the network interface from the PVE GUI, eth0 came up again, but not lo.

Could it be some edge case scenario with devuan? Are there node-side scripts that I should look for in order to investigate further?
Thanks
EDIT: additional info
 
Last edited:
Hello again.

After some digging, it looks like the network issue is a side effect of not being able to mount /run. From the node's journal:
kernel: audit: type=1400 audit(1779270973.875:168): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxc-100053_</var/lib/lxc>" name="/run/" pid=105111 comm="mount" flags="rw, move"

On the other nodes, I have no such errors.
What could cause this?
Changing title for clarity
 
Can you post the configuration of the container? Simply run pct config <CT-ID> (replace <CT-ID> with the container's ID of course), then post the result here, ideally between [code][/code] tags. Thank you!
 
sterzy said:
Can you post the configuration of the container? Simply run pct config <CT-ID> (replace <CT-ID> with the container's ID of course), then post the result here, ideally between [code][/code] tags. Thank you!
Click to expand...
Of course:
Code: 
arch: amd64
cores: 1
hostname: ns0.[redacted].org
memory: 512
net0: name=eth0,bridge=vmbr0,firewall=1,gw=[redacted],gw6=[...],hwaddr=[...],ip=[...]/32,ip6=[...]/64,type=veth
onboot: 1
ostype: devuan
rootfs: vmdata:subvol-100053-disk-0,size=8G
swap: 512
unprivileged: 1

What strikes me as odd regarding apparmor is that, in /var/lib/lxc/100053/apparmor/lxc-100053_\<-var-lib-lxc\>, there is
Code: 
# allow tmpfs mounts everywhere
mount fstype=tmpfs,
, and no other stanza anywhere seems to prohibit mounting of tmpfs on /run. The apparmor rules are exactly the same as on the guests where it works.

Thanks for taking a look :)
 
A quick update.
First of all, even though the guests are not running systemd, nesting=1 is enough to solve the issue. I am wondering why.
Second, I was under the impression that some guests worked, but it was only because it was their first boot, and the non-tmpfs /run was clean. Subsequent boots of guests also display the same unexplained behaviour, ie. apparmor forbids mounting of /run and a non-clean /run means no/buggy network.

In other words, nesting=1 looks required by devuan, but I would love to hear why, as well as why the "allow tmpfs" from the apparmor profile is not enough.

Thanks!
 
Last edited:
Thanks for the updates! I'll see if I can give you a more concrete answer once I found out a little about what's going on here.
 
I'll just add in that Debian Trixie does it too if you remove systemd (so basically the same as Devuan Excalibur). I had this issue on every single guest on my work server, and just found it on a few of my home guests.
I worked around it for guests where loopback was important by simply adding "ifdown lo" and then "ifup lo" to rc.local, but this isn't a pretty solution since it will eventually be overwritten.

I do have some guests running with nesting, so maybe the ones I've not had issues with are the nesting ones... I'll have to check that later.
 
sterzy said:
Thanks for the updates! I'll see if I can give you a more concrete answer once I found out a little about what's going on here.
Click to expand...
Hi there Shannon,

Unless I missed something elsewhere, I haven't heard anything on this issue in 2 months. Is there any way we can help diagnose/fix whatever needs fixing? I doubt that using nesting on all guests is the ideal solution, are there any side-effects to watch out for in the meantime?

Thanks!
 
Le_Coyote said:
Unless I missed something elsewhere, I haven't heard anything on this issue in 2 months. Is there any way we can help diagnose/fix whatever needs fixing? I doubt that using nesting on all guests is the ideal solution, are there any side-effects to watch out for in the meantime?
Click to expand...
Hey, yeah sorry I've been kind of swamped with other tasks recently. I'll see what I can do here, though.
 
Finally got around to this and managed to reproduce this by upgrading a Devuan 5.0 container to Devuan 6.0 via the official guide [1] on a PVE 9 node. When nesting is set we lxc enables much more permissive mounting:

Code: 
   # TODO: There doesn't seem to be a way to ask for:
   # mount options=(ro,nosuid,nodev,noexec,remount,bind),
   # as we always get mount to $cdir/proc/sys with those flags denied
   # So allow all mounts until that is straightened out:
   mount,'

Hence, mounting the tmpfs works when using nesting=1. As to why the mount is refused without that flag even though a stanza, that seemingly allows it, is present in the apparmor profile, I did some digging. After a bit of studying of man 5 apparmor.d I came across the known bugs section [2]. It states:
• The fstype may not be matched against when certain mount command flags are used. Specifically fstype matching currently only works when creating a new mount and not remount, bind, etc.
Click to expand...

Looking at the actual log output I get with my container it's the following:

Jul 27 12:32:13 host kernel: audit: type=1400 audit(1785151933.201:180): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="lxc-132_</var/lib/lxc>" name="/run/" pid=183895 comm="mount" flags="rw, move"
Click to expand...

So this is not a classical mount operation, but rather the mount is being moved from one mount point to another. So my conclusion would be that the mount fstype=tmpfs, simply does not work for moving mount points.

As for what to do for now, I think the “solution” is to use the more permissive nesting=1 and wait for apparmor to fix this bug. Sadly I found it difficult to figure out if this bug is still being actively tracked or why it seems to surface more now with apparmor 4 on Proxmox VE 9. However, I did confirm that adding lxc.apparmor.raw: mount, does remove the warnings about the mount to /run, /run/lock and /dev/shm.

I suppose we could evaluate adding specific mount rules, such as mount options=(rw,move) -> {/run/,/run/lock/,/dev/shm/}, to the apparmor profile. In my testing they also helped mount /run and the other couple of mounts apparamor complained about. If you want to test it simply add this line to your container's config:

Code: 
lxc.apparmor.raw: mount options=(rw,move) -> {/run/,/run/lock/,/dev/shm/},

If you think we should consider adding such a line, I'd suggest you open a bug on our Bugzilla [3]. I am too unfamiliar with apparmor to feel confident in making the judgement call of whether such an addition is reasonable or too risky.

Hope that helps!

[1]: https://www.devuan.org/os/documentation/install-guides/excalibur/upgrade-to-excalibur
[2]: https://apparmor.net/man/5.0/apparmor.d/#KNOWN-BUGS
[3]: https://bugzilla.proxmox.com/
 
Hello Shannon,

Thank you so much for digging into this. Being far less knowledgeable about apparmor, I wouldn't have made it this far.

I can confirm that:
sterzy said:
lxc.apparmor.raw: mount options=(rw,move) -> {/run/,/run/lock/,/dev/shm/},
Click to expand...
makes my Devuan guests start as expected, ie. no more network interface issue. I prefer that approach to the nesting one, seeing as the latter increases the attack surface a bit, and the recent string of LPE exploits has made me somewhat nervous (if not paranoid).

For completeness' sake, I would have expected this fix to leave the guest's /run completely empty after shutdown, but that is not entirely the case. When stopped, it contains the following:
Code: 
prw------- 1 100000 100000   0 Jul 29 13:30 initctl
drwxr-xr-x 2 100000 100000   2 Jul 29 13:30 mount
-rw-r--r-- 1 100000 100000 384 Jul 29 13:31 utmp
(and "mount" itself is empty).
This does not seem to adversely impact the container, but I am unsure what to make of it. Any ideas?

Thanks a lot for your help,
Regards
--
Romain
 
I can't directly reproduce this I am sad to say, but my assumption would be that some services write to /run before the move mount succeeds. Hence, the files are simply written to the guests root file system. All the fix does is generally allow the move mount, it does not clean up files that were previously written there.

Note that once the move mount succeeds, the mount makes the underlying data at the same location inaccessible.
 
sterzy said:
I can't directly reproduce this I am sad to say, but my assumption would be that some services write to /run before the move mount succeeds. Hence, the files are simply written to the guests root file system. All the fix does is generally allow the move mount, it does not clean up files that were previously written there.

Note that once the move mount succeeds, the mount makes the underlying data at the same location inaccessible.
Click to expand...
Of course, come to think of it, as long as the mount succeeds, this should not be too big of a problem. Thank you for the heads-up.

Regarding your earlier suggestion to open a bug, I'll do so right now, if only to make sure the issue stays on the radar.

Thanks again!

Edit: link to bug report https://bugzilla.proxmox.com/show_bug.cgi?id=7862
 
Last edited:
  • Like
Reactions: sterzy
You must log in or register to reply here.
Top Bottom
Back