Unable to boot lxc after update to PVE 6.2 - Failed to mount "run" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/run"

[cluster-aware]

I have two lxc containers that are no longer starting after upgrading to PVE 6.2 and rebooting.

PVE manager shows the start task, then nothing. Adding debug output to lxc-start reveals Failed to mount "run" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/run"

Creating a new CT on the same host boots ok.

lxc-start -n 107 -F -l DEBUG -o ~/debug-log

lxc-start 107 20200711155550.468 ERROR    conf - conf.c:mount_entry:1855 - No such file or directory - Failed to mount "run" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/run"
lxc-start 107 20200711155550.468 ERROR    conf - conf.c:lxc_setup:3329 - Failed to setup mount entries
lxc-start 107 20200711155550.468 ERROR    start - start.c:do_start:1231 - Failed to setup container "107"
lxc-start 107 20200711155550.468 ERROR    sync - sync.c:__sync_wait:41 - An error occurred in another process (expected sequence number 5)
lxc-start 107 20200711155550.468 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:852 - Received container state "ABORTING" instead of "RUNNING"
lxc-start 107 20200711155550.468 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 107 20200711155550.468 ERROR    lxc_start - tools/lxc_start.c:main:311 - To get more details, run the container in foreground mode
lxc-start 107 20200711155550.468 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start 107 20200711155550.468 ERROR    start - start.c:__lxc_start:1957 - Failed to spawn container "107"
....

Spoiler: pveversion

proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-6 (running version: 6.2-6/ee1d7754)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.0: 6.0-11
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-4.15: 5.4-8
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-1-pve: 5.0.21-2
pve-kernel-4.15.18-20-pve: 4.15.18-46
pve-kernel-4.15.18-12-pve: 4.15.18-36
pve-kernel-4.15.18-10-pve: 4.15.18-32
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-3
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
openvswitch-switch: 2.12.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-8
pve-cluster: 6.1-8
pve-container: 3.1-8
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-4
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-3
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

Spoiler: /var/lib/lxc/107/config

lxc.cgroup.relative = 0
lxc.cgroup.dir.monitor = lxc.monitor/107
lxc.cgroup.dir.container = lxc/107
lxc.cgroup.dir.container.inner = ns
lxc.arch = amd64
lxc.include = /usr/share/lxc/config/alpine.common.conf
lxc.include = /usr/share/lxc/config/alpine.userns.conf
lxc.seccomp.profile = /usr/share/lxc/config/pve-userns.seccomp
lxc.apparmor.profile = generated
lxc.apparmor.raw = deny mount -> /proc/,
lxc.apparmor.raw = deny mount -> /sys/,
lxc.mount.auto = sys:mixed
lxc.monitor.unshare = 1
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.tty.max = 2
lxc.environment = TERM=linux
lxc.uts.name = dhcp
lxc.cgroup.memory.limit_in_bytes = 33554432
lxc.cgroup.memory.memsw.limit_in_bytes = 67108864
lxc.cgroup.cpu.shares = 1024
lxc.rootfs.path = /var/lib/lxc/107/rootfs
lxc.net.0.type = veth
lxc.net.0.veth.pair = veth107i0
lxc.net.0.hwaddr = 92:3E:57:B5:5C:E3
lxc.net.0.name = eth0
lxc.net.0.script.up = /usr/share/lxc/lxcnetaddbr
lxc.cgroup.cpuset.cpus = 17

 

[fiona]

Hi,
please try mounting the container with pct mount 107, go to the mountpoint and verify that the the expected data is still there. In particular there should be an empty run directory in /var/lib/lxc/107/rootfs/. To unmount the filesystem(s) afterwards, use pct unmount 107.