LCX not starting after upgrading PVe from 9.1.x to 9.2.2 - Mount fails

E

elendiir

Member
Feb 1, 2022
15
0
21
56
Hi,
I just have upgraded our PVE environment from 9.1.x (not sure which version it has been) to 9.2.2. using the productive repro. PVE-Container version is 6.1.10
Now I am not able to start any LXCs anymore.

Starting the LCX in debug shows me the following errors:
Code: 
lxc-start 501 20260612093837.936 DEBUG    utils - ../src/lxc/utils.c:run_buffer:558 - Script exec /usr/share/lxcfs/lxc.mount.hook '501' 'lxc' 'mount'
produced output: mount: /usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/cpuinfo: bind /var/lib/lxcfs/proc/cpuinfo failed.
       dmesg(1) may have more information after failed mount system call.
lxc-start 501 20260612093837.936 ERROR    utils - ../src/lxc/utils.c:run_buffer:569 - Script exited with status 32
lxc-start 501 20260612093837.936 ERROR    conf - ../src/lxc/conf.c:lxc_setup:3845 - Failed to run mount hooks
lxc-start 501 20260612093837.936 ERROR    start - ../src/lxc/start.c:do_start:1466 - Failed to setup container "501"
lxc-start 501 20260612093837.936 ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc-start 501 20260612093837.936 DEBUG    network - ../src/lxc/network.c:lxc_delete_network:4493 - Deleted network devices

This is one example LCX.conf:
Code: 
arch: amd64
cores: 2
features: keyctl=1,nesting=1
hostname: pihole
memory: 2048
net0: name=eth0,bridge=vmbr0,gw=10.0.0.254,hwaddr=BC:24:11:39:41:85,ip=10.0.0.8/24,ip6=auto,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-501-disk-0,size=10G
startup: order=1,up=10,down=10
swap: 512
tags: 
unprivileged: 1

The LCXs have been working fine before the update. It happens on all our 3 servers.

Any idea what to do? I have found older threads having a similar issue. But all of those seems to be fixed with 9.1.x already.

Thanks
Fritz
 
Sorry I forgot to post this part. I am not sure what part is of interest. I think this line might show the error?
Code: 
[Jun12 13:24] audit: type=1400 audit(1781263485.458:15832): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-st>
[  +0.051371] audit: type=1400 audit(1781263485.510:15833): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-st>
[  +0.064925] EXT4-fs (dm-6): mounted filesystem 2fe804d6-8a10-4df5-b260-b496f8a968ed r/w with ordered data mode. Quota mode: none.
[  +0.233953] audit: type=1400 audit(1781263485.808:15834): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-5>
[  +0.494820] vmbr0: port 2(veth501i0) entered blocking state
[  +0.000006] vmbr0: port 2(veth501i0) entered disabled state
[  +0.000027] veth501i0: entered allmulticast mode
[  +0.000052] veth501i0: entered promiscuous mode
[  +0.036034] eth0: renamed from vethojCVBk
[  +0.125635] audit: type=1400 audit(1781263486.465:15835): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc>
[  +0.021103] vmbr0: port 2(veth501i0) entered disabled state
[  +0.000210] veth501i0 (unregistering): left allmulticast mode
[  +0.000004] veth501i0 (unregistering): left promiscuous mode
[  +0.000004] vmbr0: port 2(veth501i0) entered disabled state
[  +0.421669] EXT4-fs (dm-6): unmounting filesystem 2fe804d6-8a10-4df5-b260-b496f8a968ed.
[Jun12 13:26] audit: type=1400 audit(1781263597.426:15836): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-st>
[  +0.055083] audit: type=1400 audit(1781263597.481:15837): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-st>
[  +0.075805] EXT4-fs (dm-6): mounted filesystem 2fe804d6-8a10-4df5-b260-b496f8a968ed r/w with ordered data mode. Quota mode: none.
[  +0.218860] audit: type=1400 audit(1781263597.776:15838): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-5>
[  +0.496341] vmbr0: port 2(veth501i0) entered blocking state
[  +0.000007] vmbr0: port 2(veth501i0) entered disabled state
[  +0.000028] veth501i0: entered allmulticast mode
[  +0.000057] veth501i0: entered promiscuous mode
[  +0.040510] eth0: renamed from veth8dD3mA
[  +0.123692] audit: type=1400 audit(1781263598.437:15839): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc>
[  +0.021143] vmbr0: port 2(veth501i0) entered disabled state
[  +0.000212] veth501i0 (unregistering): left allmulticast mode
[  +0.000003] veth501i0 (unregistering): left promiscuous mode
[  +0.000003] vmbr0: port 2(veth501i0) entered disabled state
[  +0.435085] EXT4-fs (dm-6): unmounting filesystem 2fe804d6-8a10-4df5-b260-b496f8a968ed.
 
Last edited:
Does apt -U dist-upgrade go through without warnings/errors? What's systemctl --failed say? Does it behave/look different after a reboot of the node?
 
Good morning,

apt -U dist-upgrade works fine. No errors. I just have updated librabbitmq4.

systemctl --failed shows me those two LXCs that I tried to restart on this server:
Code: 
root@proxmox-2:~# systemctl --failed
  UNIT                      LOAD   ACTIVE SUB    DESCRIPTION           
â—Ź pve-container@501.service loaded failed failed PVE LXC Container: 501
â—Ź pve-container@503.service loaded failed failed PVE LXC Container: 503

I will try o reboot tomorrow morning. I need to move all VMs to the other two servers. And I need to be in the office for the reboot. Since this server is hosting our VPN/Firewall aplliance.

Any other idea that I could try before the reboot?
 
elendiir said:
Sorry I forgot to post this part. I am not sure what part is of interest. I think this line might show the error?
Click to expand...
Have you started the affected container during the time of this log? You can use journalctl --since '2026-06-12 09:38:00' --until '...' to capture the output between some time span (I filled in the --since with the rough start time of the LXC from your top post). Adding --no-pager might help in copying the output, so that longer lines (like the apparmor messages) are not truncated.
 
Thanks Daniel for providing the correct command syntax.

Here is the correct output from journalctl that does contain the failed start of LXC. Just the clearify: it is not only this container that fails. I see all containers not working on our 3 Proxmox-servers after installing the update to 9.2.2.

Code: 
Jun 15 09:50:18 proxmox-2 pvedaemon[2877409]: <root@pam> starting task UPID:proxmox-2:002BE96E:1108C062:6A2FAEBA:vzstart:501:root@pam:
Jun 15 09:50:18 proxmox-2 pvedaemon[2877806]: starting CT 501: UPID:proxmox-2:002BE96E:1108C062:6A2FAEBA:vzstart:501:root@pam:
Jun 15 09:50:19 proxmox-2 systemd[1]: Started pve-container@501.service - PVE LXC Container: 501.
Jun 15 09:50:19 proxmox-2 kernel: audit: type=1400 audit(1781509819.574:15840): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2877815 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Jun 15 09:50:19 proxmox-2 kernel: audit: type=1400 audit(1781509819.624:15841): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2877816 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Jun 15 09:50:19 proxmox-2 kernel: EXT4-fs (dm-6): mounted filesystem 2fe804d6-8a10-4df5-b260-b496f8a968ed r/w with ordered data mode. Quota mode: none.
Jun 15 09:50:19 proxmox-2 dbus-daemon[1091]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service' requested by ':1.252175' (uid=0 pid=2877823 comm="timedatectl show --property=Timezone --value" label="/usr/bin/lxc-start (enforce)")
Jun 15 09:50:19 proxmox-2 systemd[1]: Starting systemd-timedated.service - Time & Date Service...
Jun 15 09:50:19 proxmox-2 systemd[1]: Started systemd-timedated.service - Time & Date Service.
Jun 15 09:50:19 proxmox-2 dbus-daemon[1091]: [system] Successfully activated service 'org.freedesktop.timedate1'
Jun 15 09:50:19 proxmox-2 kernel: audit: type=1400 audit(1781509819.922:15842): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-501_</var/lib/lxc>" pid=2877832 comm="apparmor_parser"
Jun 15 09:50:20 proxmox-2 kernel: vmbr0: port 2(veth501i0) entered blocking state
Jun 15 09:50:20 proxmox-2 kernel: vmbr0: port 2(veth501i0) entered disabled state
Jun 15 09:50:20 proxmox-2 kernel: veth501i0: entered allmulticast mode
Jun 15 09:50:20 proxmox-2 kernel: veth501i0: entered promiscuous mode
Jun 15 09:50:20 proxmox-2 kernel: eth0: renamed from vethyQPIky
Jun 15 09:50:20 proxmox-2 lxcfs[1946804]: ../src/proc_fuse.c: 160: proc_getattr: Due to restricted personality access policy, reading proc files from containers is not permitted
Jun 15 09:50:20 proxmox-2 lxcfs[1946804]: ../src/proc_fuse.c: 160: proc_getattr: Due to restricted personality access policy, reading proc files from containers is not permitted
Jun 15 09:50:20 proxmox-2 pvedaemon[2877806]: startup for container '501' failed
Jun 15 09:50:20 proxmox-2 kernel: audit: type=1400 audit(1781509820.618:15843): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-501_</var/lib/lxc>" pid=2877935 comm="apparmor_parser"
Jun 15 09:50:20 proxmox-2 pvedaemon[2877409]: unable to get PID for CT 501 (not running?)
Jun 15 09:50:20 proxmox-2 kernel: vmbr0: port 2(veth501i0) entered disabled state
Jun 15 09:50:20 proxmox-2 kernel: veth501i0 (unregistering): left allmulticast mode
Jun 15 09:50:20 proxmox-2 kernel: veth501i0 (unregistering): left promiscuous mode
Jun 15 09:50:20 proxmox-2 kernel: vmbr0: port 2(veth501i0) entered disabled state
Jun 15 09:50:20 proxmox-2 pvedaemon[2877409]: <root@pam> end task UPID:proxmox-2:002BE96E:1108C062:6A2FAEBA:vzstart:501:root@pam: startup for container '501' failed
Jun 15 09:50:21 proxmox-2 kernel: EXT4-fs (dm-6): unmounting filesystem 2fe804d6-8a10-4df5-b260-b496f8a968ed.
Jun 15 09:50:21 proxmox-2 systemd[1]: pve-container@501.service: Main process exited, code=exited, status=1/FAILURE
Jun 15 09:50:21 proxmox-2 systemd[1]: pve-container@501.service: Failed with result 'exit-code'.
Jun 15 09:50:21 proxmox-2 systemd[1]: pve-container@501.service: Consumed 643ms CPU time, 93.3M memory peak.

Any help or ideas are more than welcome!
 
are you using the "YAMA" LSM or have restricted ptrace access globally?

could you post the output of "sysctl kernel.yama.ptrace_scope"?
 
Hm,

not sure if this setting has been touched. Here is the wanted output:
Code: 
root@proxmox-2:~# sysctl kernel.yama.ptrace_scope
kernel.yama.ptrace_scope = 3
 
it might have been deployed in a prior boot as mitigation for one of the recent kernel security issues? it seems there have been issues in the past in lxcfs with it that might not have been completely fixed..
 
As far as I know the only changes that have been made can be found in /etc/modprobe.de:
One is dirty-frag.conf
Code: 
install esp4 /bin/false
install esp6 /bin/false
install rxrpc /bin/false

And for copy-fail called disable-algif.conf
Code: 
install algif_aead /bin/false
 
well, the value is not the default so it must be set somewhere (kernel cmdline? some custom script/service running at boot?)
 
So now I do understand you. The non-default kernel.yama.ptrace_scope = 3 might cause the issues. Now we need to understand why it is not set to 1 which I assume should be the default. At least I see kernel.yama.ptrace_scope = 1 on my home-lab environment.
 
yes, sorry for not writing that clearly. I can reproduce failing container starts with 3, everything up to 2 seems to work!
 
Something like this might reveal if/where it is configured
Bash: 
grep -sR "ptrace_scope" /etc
There might also be a command in history containing it.
 
Last edited:
@fabian: no worries. I am glad that we have found the reason for the non-starting LXCs. Even when I still not understand how and why the value has been changed to 3.

@Impact: no files found

I theory I should be able to change the value on the running server using:
echo 1 | sudo tee /proc/sys/kernel/yama.ptrace_scope

And now the start of the LXCs should already work, correct? Does this have any other affects on e.g. running VMs?
 
it's not possible to go back from 3 unfortunately (without a reboot, which in turn requires finding out what sets it at boot).
 
you could also check /lib/sysctl.d or /usr/local/lib/sysctl.d
 
You must log in or register to reply here.
Top Bottom
Back