LXC containers do not start after upgrade to Proxmox 8.2.

gim

New Member
Aug 18, 2024
3
0
1
Hi all,

I upgraded from Proxmox 7 to 8 today and everything worked without showing any errors. VMs are starting and running as expected but LXC container do not start (tried also a completely new container but same behavior)

  • I get the following error trying to start LXC containers

Code:
sync_wait: 34 An error occurred in another process (expected sequence number 4)
__lxc_start: 2114 Failed to spawn container "100"
TASK ERROR: startup for container '100' failed

  • Starting the container with loglevel DEBUG showed the following ERRORS
Code:
pct start 100 --debug
...

INFO     conf - ../src/lxc/conf.c:lxc_setup_ttys:876 - Finished setting up 2 /dev/tty<N> device(s)
ERROR    conf - ../src/lxc/conf.c:lxc_pivot_root:1453 - Permission denied - Failed to turn new root mount tree into shared mount tree
ERROR    conf - ../src/lxc/conf.c:lxc_setup:3986 - Failed to pivot root into rootfs
ERROR    start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "100"
ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
DEBUG    network - ../src/lxc/network.c:lxc_delete_network:4217 - Deleted network devices
ERROR    start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "100"
WARN     start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 16 for process 117375
startup for container '100' failed

I already googled the error messages but I haven't found a solution yet.


  • My PVE version
Code:
pveversion -v

proxmox-ve: 8.2.0 (running kernel: 5.15.102-1-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-15
proxmox-kernel-6.8: 6.8.12-1
proxmox-kernel-6.8.12-1-pve-signed: 6.8.12-1
pve-kernel-5.15.158-2-pve: 5.15.158-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx9
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.2
libpve-guest-common-perl: 5.1.4
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.5.0
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.3
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.13-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.2-2
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

  • Running the pve7to8 tool again shows 2 Warnings

Code:
pve7to8 --full
= CHECKING VERSION INFORMATION FOR PVE PACKAGES =

Checking for package updates..
PASS: all packages up-to-date

Checking proxmox-ve package version..
PASS: already upgraded to Proxmox VE 8

Checking running kernel version..
WARN: unexpected running and installed kernel '5.15.102-1-pve'.

= CHECKING CLUSTER HEALTH/SETTINGS =

SKIP: standalone node.

= CHECKING HYPER-CONVERGED CEPH STATUS =

SKIP: no hyper-converged ceph setup detected!

= CHECKING CONFIGURED STORAGES =

PASS: storage 'local' enabled and active.
PASS: storage 'local-zfs' enabled and active.
PASS: storage 'neptun_backup_proxmox' enabled and active.
INFO: Checking storage content type configuration..
PASS: no storage content problems found
PASS: no storage re-uses a directory for multiple content types.

= MISCELLANEOUS CHECKS =

INFO: Checking common daemon services..
PASS: systemd unit 'pveproxy.service' is in state 'active'
PASS: systemd unit 'pvedaemon.service' is in state 'active'
PASS: systemd unit 'pvescheduler.service' is in state 'active'
PASS: systemd unit 'pvestatd.service' is in state 'active'
INFO: Checking for supported & active NTP service..
PASS: Detected active time synchronisation unit 'chrony.service'
INFO: Checking for running guests..
WARN: 2 running guest(s) detected - consider migrating or stopping them.
INFO: Checking if the local node's hostname 'morpheus' is resolvable..
INFO: Checking if resolved IP is configured on local node..
PASS: Resolved node IP '192.168.3.3' configured and active on single interface.
INFO: Check node certificate's RSA key size
PASS: Certificate 'pve-root-ca.pem' passed Debian Busters (and newer) security level for TLS connections (4096 >= 2048)
PASS: Certificate 'pve-ssl.pem' passed Debian Busters (and newer) security level for TLS connections (2048 >= 2048)
PASS: Certificate 'pveproxy-ssl.pem' passed Debian Busters (and newer) security level for TLS connections (4096 >= 2048)
INFO: Checking backup retention settings..
PASS: no backup retention problems found.
INFO: checking CIFS credential location..
PASS: no CIFS credentials at outdated location found.
INFO: Checking permission system changes..
INFO: Checking custom role IDs for clashes with new 'PVE' namespace..
PASS: no custom roles defined, so no clash with 'PVE' role ID namespace enforced in Proxmox VE 8
INFO: Checking if LXCFS is running with FUSE3 library, if already upgraded..
PASS: systems seems to be upgraded and LXCFS is running with FUSE 3 library
INFO: Checking node and guest description/note length..
PASS: All node config descriptions fit in the new limit of 64 KiB
PASS: All guest config descriptions fit in the new limit of 8 KiB
INFO: Checking container configs for deprecated lxc.cgroup entries
PASS: No legacy 'lxc.cgroup' keys found.
INFO: Checking if the suite for the Debian security repository is correct..
PASS: found no suite mismatch
INFO: Checking for existence of NVIDIA vGPU Manager..
PASS: No NVIDIA vGPU Service found.
INFO: Checking bootloader configuration...
PASS: bootloader packages installed correctly
INFO: Check for dkms modules...
SKIP: could not get dkms status

= SUMMARY =

TOTAL:    31
PASSED:   26
SKIPPED:  3
WARNINGS: 2
FAILURES: 0

ATTENTION: Please check the output for detailed information!

Does anyone have an idea how to fix the Permission denied error and being able to turn new root mount tree to a shared one?
 
Last edited:
proxmox-ve: 8.2.0 (running kernel: 5.15.102-1-pve)

Did you pin that kernel?
If yes, what is/was the reason?
If no, you should reboot the PVE-host to boot with the new kernel and also is, what you generally should do after a major upgrade.
 
Thanks for your reply. I did not pin the kernel and restarted the host multiple times.

I actually pinned the new kernel but it did not had an effect.

Code:
proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
5.15.158-2-pve
6.8.12-1-pve

Pinned kernel:
6.8.12-1-pve

In the meantime I managed to boot the system with the new kernel but the LXC containers still do not start, the error message is the same.

Code:
uname -a
Linux morpheus 6.8.12-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.12-1 (2024-08-05T16:17Z) x86_64 GNU/Linux


What I have also seen in the syslog log are errors from apparmor when trying to start the LXC containers

Code:
2024-08-18T16:30:01.780389+02:00 xxxx pve-guests[2001]: startup for container '100' failed
2024-08-18T16:30:01.782375+02:00 xxxx kernel: [   15.194459] audit: type=1400 audit(1723991401.777:29): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="/usr/bin/lxc-start" name="/" pid=2023 comm="lxc-start" flags="rw, rshared"
...
Aug 18 00:00:45 xxxxx kernel: [ 6594.596546] audit: type=1400 audit(1723932045.610:3386): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-101_</var/lib/lxc>" name="/run/systemd/unit-root/proc/" pid=601107 comm="(d-logind)" fstype="proc" srcname="proc" flags="rw, nosuid, nodev, noexec"
...
 
Last edited:
I just installed Proxmox 8.2.4 from scratch and currently I am restoring my vms and containers.
Already checked if lxc containers and VMs are starting now and it worked perfectly fine.

Still see the apparmor errors in the syslog log, but it seems that this does not have any impact.
 
running dmesg --time-format=iso | grep -i error seeing

Code:
apparmor="DENIED" operation="mount" class="mount" info="failed perms check" error=-13 profile="lxc-102_</var/lib/lxc>"

This has also prevented containers starting.
Full log attached.
 

Attachments

  • pve_log.txt
    69 KB · Views: 0
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!