LXC Containers Not Starting After Update

nintendo424

New Member
Jul 21, 2021
8
1
3
29
Hi, I recently updated my Proxmox VE instance and after rebooting to use the new kernel, my LXC Unprivileged containers are no longer working. For example, I'm receiving the following errors:

Bash:
pct start 100
safe_mount: 1200 Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
lxc_mount_auto_mounts: 810 Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
lxc_setup: 4356 Failed to setup first automatic mounts
do_start: 1274 Failed to setup container "100"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2068 Failed to spawn container "100"
startup for container '100' failed

Output of config:
Bash:
pct config 100
arch: amd64
cores: 4
features: nesting=1
hostname: SteamServerDebian
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=52:54:00:45:f8:9d,ip=dhcp,type=veth
onboot: 0
ostype: debian
rootfs: VMs:100/vm-100-disk-0.raw,size=100G
swap: 16384
unprivileged: 1

This was working fine prior to upgrading. I determined that creating a new privileged VM fixes the issue, but that seems like a workaround instead of a fix. Did something change with LXC configurations?

pveversion -v output:
Bash:
pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-8 (running version: 7.1-8/5b267f33)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-4
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
 

vmware

New Member
Jul 8, 2020
29
8
3
Sydney, Australia
Hi all,

I've got the exact same issue after apt upgrade 30 minutes ago. Unprivileged containers don't boot anymore.

Code:
safe_mount: 1200 Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
lxc_mount_auto_mounts: 810 Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
lxc_setup: 4356 Failed to setup first automatic mounts
do_start: 1274 Failed to setup container "103"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2068 Failed to spawn container "103"
startup for container '103' failed
 

vmware

New Member
Jul 8, 2020
29
8
3
Sydney, Australia
Starting the unprivileged LXC with --debug option:

Code:
INFO     network - network.c:lxc_setup_network_in_child_namespaces:4005 - Finished setting up network devices with caller assigned names
INFO     conf - conf.c:mount_autodev:1215 - Preparing "/dev"
INFO     conf - conf.c:mount_autodev:1276 - Prepared "/dev"
DEBUG    conf - conf.c:lxc_mount_auto_mounts:735 - Invalid argument - Tried to ensure procfs is unmounted
DEBUG    conf - conf.c:lxc_mount_auto_mounts:758 - Invalid argument - Tried to ensure sysfs is unmounted
ERROR    utils - utils.c:safe_mount:1200 - Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
ERROR    conf - conf.c:lxc_mount_auto_mounts:810 - Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
ERROR    conf - conf.c:lxc_setup:4356 - Failed to setup first automatic mounts
ERROR    start - start.c:do_start:1274 - Failed to setup container "103"
ERROR    sync - sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
DEBUG    network - network.c:lxc_delete_network:4159 - Deleted network devices
ERROR    start - start.c:__lxc_start:2068 - Failed to spawn container "103"
WARN     start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 16 for process 220755
startup for container '103' failed
 

lps90

Member
May 21, 2020
168
6
23
My VM's are with problems too after updating:

"Dec 13 20:38:25 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:38:44 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:39:03 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:39:22 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:39:41 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:00 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:19 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:38 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:57 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:41:16 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:41:35 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:41:54 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:42:13 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:42:32 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:42:51 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:43:10 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout"
 

vmware

New Member
Jul 8, 2020
29
8
3
Sydney, Australia
Brand new created unprivileged container is not working, created from official debian 11.0 template:

Code:
arch: amd64
cores: 2
features: nesting=1
hostname: filez-new
memory: 2048
nameserver: 192.168.1.1
net0: name=eth0,bridge=vmbr30,firewall=1,hwaddr=96:D2:62:33:2E:39,ip=dhcp,ip6=auto,type=veth
ostype: debian
rootfs: local-lvm:vm-313-disk-0,mountoptions=lazytime;noatime,size=4G
swap: 2048
unprivileged: 1
 
  • Like
Reactions: lps90

lps90

Member
May 21, 2020
168
6
23
Brand new created unprivileged container is not working, created from official debian 11.0 template:

Code:
arch: amd64
cores: 2
features: nesting=1
hostname: filez-new
memory: 2048
nameserver: 192.168.1.1
net0: name=eth0,bridge=vmbr30,firewall=1,hwaddr=96:D2:62:33:2E:39,ip=dhcp,ip6=auto,type=veth
ostype: debian
rootfs: local-lvm:vm-313-disk-0,mountoptions=lazytime;noatime,size=4G
swap: 2048
unprivileged: 1

Confirmed!
The console is not even showing..
"failed waiting for client: timed out
TASK ERROR: command '/usr/bin/termproxy 5900 --path /vms/101 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole101 -r winch -z lxc-console -n 101 -e -1' failed: exit code 1 "
 
Last edited:

vmware

New Member
Jul 8, 2020
29
8
3
Sydney, Australia
I've tested creating a new LXC CT on an Intel machine running PVE 7.1-8. It's been upgraded to latest Proxmox no-subscription release, but has not been reboot (Not sure if reboot is a safe option at this stage). Running 5.11.22 kernel.

Interestingly the Intel machine is not affected by this issue. My AMD machine is affected.
 

vmware

New Member
Jul 8, 2020
29
8
3
Sydney, Australia
Managed to fix the issue. Now my CTs are once again starting normally.

In my /etc/fstab I customised the /proc filesystem with the following line:

Code:
proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=1 0 0

By reverting the line to Proxmox's default, then reboot, the problem is solved:

Code:
proc /proc proc defaults 0 0

Phew!
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,832
1,034
164
Brand new created unprivileged container is not working, created from official debian 11.0 template:
tried that here - without any modifications - and cannot reproduce the issue.

In my /etc/fstab I customised the /proc filesystem with the following line:
this seems like the likely reason for the problems - congrats on finding this so fast!
a default container usually does not have any line in /etc/fstab (apart from a comment) - did you modify /proc on your PVE-node?

@nintendo424 - just for confirmation - do you also have custom options in fstab for proc?
 

lps90

Member
May 21, 2020
168
6
23
Managed to fix the issue. Now my CTs are once again starting normally.

In my /etc/fstab I customised the /proc filesystem with the following line:

Code:
proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=1 0 0

By reverting the line to Proxmox's default, then reboot, the problem is solved:

Code:
proc /proc proc defaults 0 0

Phew!

I'm sorry but i didnt understand.
what is the one we need to put in fstab file?

" proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=1 0 0 "
"proc /proc proc defaults 0 0 "
 

lps90

Member
May 21, 2020
168
6
23
Ok ;)
I just formated my machine with Proxmox v7.1-4 to solve all the problems caused by the latest v7.1-8 version.
Currently the server machine is not randomly rebooting again and all LXC and VM's are working with no problems.
 

nintendo424

New Member
Jul 21, 2021
8
1
3
29
tried that here - without any modifications - and cannot reproduce the issue.


this seems like the likely reason for the problems - congrats on finding this so fast!
a default container usually does not have any line in /etc/fstab (apart from a comment) - did you modify /proc on your PVE-node?

@nintendo424 - just for confirmation - do you also have custom options in fstab for proc?
I do not have custom options in my fstab for /proc, and my LXC containers were also brand new ones based on the Debian template. I will double check my fstab when I get a chance and post back. It does sound like this is the fix though, just not sure what would have changed the fstab entry.
 

nintendo424

New Member
Jul 21, 2021
8
1
3
29
Checking my fstab entry, it was in fact different. It said
Bash:
proc /proc proc defaults,noatime 0 0

I updated it to just have defaults and sure enough, unprivileged containers are fine now.
 

Stoiko Ivanov

Proxmox Staff Member
Staff member
May 2, 2018
6,832
1,034
164
Glad that worked out for you as well!
 

shamahn

New Member
Jul 16, 2021
4
0
1
41
panic :eek:
Same problem: after updating the host, none of the CT start.
Error:
Code:
safe_mount: 1200 Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
lxc_mount_auto_mounts: 810 Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
lxc_setup: 4356 Failed to setup first automatic mounts
do_start: 1274 Failed to setup container "100"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2068 Failed to spawn container "100"
TASK ERROR: startup for container '100' failed

But mount options are default. /etc/fstab :
Code:
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/rpool/data/subvol-101-disk-0/etc/fstab :
Code:
# UNCONFIGURED FSTAB FOR BASE SYSTEM

Brand new created unprivileged container is not working, created from official debian 11.0 template

What to do? need help!
thnx

PS. All VMs are working fine.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!