LXC Containers Not Starting After Update

N

nintendo424

New Member
Jul 21, 2021
8
1
3
31
Hi, I recently updated my Proxmox VE instance and after rebooting to use the new kernel, my LXC Unprivileged containers are no longer working. For example, I'm receiving the following errors:

Bash: 
pct start 100
safe_mount: 1200 Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
lxc_mount_auto_mounts: 810 Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
lxc_setup: 4356 Failed to setup first automatic mounts
do_start: 1274 Failed to setup container "100"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2068 Failed to spawn container "100"
startup for container '100' failed

Output of config:
Bash: 
pct config 100
arch: amd64
cores: 4
features: nesting=1
hostname: SteamServerDebian
memory: 16384
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=52:54:00:45:f8:9d,ip=dhcp,type=veth
onboot: 0
ostype: debian
rootfs: VMs:100/vm-100-disk-0.raw,size=100G
swap: 16384
unprivileged: 1

This was working fine prior to upgrading. I determined that creating a new privileged VM fixes the issue, but that seems like a workaround instead of a fix. Did something change with LXC configurations?

pveversion -v output:
Bash: 
pveversion -v
proxmox-ve: 7.1-1 (running kernel: 5.13.19-2-pve)
pve-manager: 7.1-8 (running version: 7.1-8/5b267f33)
pve-kernel-helper: 7.1-6
pve-kernel-5.13: 7.1-5
pve-kernel-5.11: 7.0-10
pve-kernel-5.4: 6.4-4
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.4.124-1-pve: 5.4.124-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-14
libpve-guest-common-perl: 4.0-3
libpve-http-server-perl: 4.0-4
libpve-storage-perl: 7.0-15
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.1.2-1
proxmox-backup-file-restore: 2.1.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-4
pve-cluster: 7.1-2
pve-container: 4.1-3
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-3
pve-ha-manager: 3.3-1
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.0-3
pve-xtermjs: 4.12.0-1
qemu-server: 7.1-4
smartmontools: 7.2-pve2
spiceterm: 3.2-2
swtpm: 0.7.0~rc1+2
vncterm: 1.7-1
zfsutils-linux: 2.1.1-pve3
 
Hi all,

I've got the exact same issue after apt upgrade 30 minutes ago. Unprivileged containers don't boot anymore.

Code: 
safe_mount: 1200 Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
lxc_mount_auto_mounts: 810 Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
lxc_setup: 4356 Failed to setup first automatic mounts
do_start: 1274 Failed to setup container "103"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2068 Failed to spawn container "103"
startup for container '103' failed
 
Starting the unprivileged LXC with --debug option:

Code: 
INFO     network - network.c:lxc_setup_network_in_child_namespaces:4005 - Finished setting up network devices with caller assigned names
INFO     conf - conf.c:mount_autodev:1215 - Preparing "/dev"
INFO     conf - conf.c:mount_autodev:1276 - Prepared "/dev"
DEBUG    conf - conf.c:lxc_mount_auto_mounts:735 - Invalid argument - Tried to ensure procfs is unmounted
DEBUG    conf - conf.c:lxc_mount_auto_mounts:758 - Invalid argument - Tried to ensure sysfs is unmounted
ERROR    utils - utils.c:safe_mount:1200 - Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
ERROR    conf - conf.c:lxc_mount_auto_mounts:810 - Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
ERROR    conf - conf.c:lxc_setup:4356 - Failed to setup first automatic mounts
ERROR    start - start.c:do_start:1274 - Failed to setup container "103"
ERROR    sync - sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
DEBUG    network - network.c:lxc_delete_network:4159 - Deleted network devices
ERROR    start - start.c:__lxc_start:2068 - Failed to spawn container "103"
WARN     start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 16 for process 220755
startup for container '103' failed
 
My VM's are with problems too after updating:

"Dec 13 20:38:25 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:38:44 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:39:03 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:39:22 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:39:41 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:00 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:19 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:38 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:40:57 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:41:16 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:41:35 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:41:54 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:42:13 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:42:32 Server pvedaemon[1390]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:42:51 Server pvedaemon[1389]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout
Dec 13 20:43:10 Server pvedaemon[1388]: VM 1000 qmp command failed - VM 1000 qmp command 'guest-ping' failed - got timeout"
 
Brand new created unprivileged container is not working, created from official debian 11.0 template:

Code: 
arch: amd64
cores: 2
features: nesting=1
hostname: filez-new
memory: 2048
nameserver: 192.168.1.1
net0: name=eth0,bridge=vmbr30,firewall=1,hwaddr=96:D2:62:33:2E:39,ip=dhcp,ip6=auto,type=veth
ostype: debian
rootfs: local-lvm:vm-313-disk-0,mountoptions=lazytime;noatime,size=4G
swap: 2048
unprivileged: 1
 
  • Like
Reactions: lps90
vmware said:
Brand new created unprivileged container is not working, created from official debian 11.0 template:

Code: 
arch: amd64
cores: 2
features: nesting=1
hostname: filez-new
memory: 2048
nameserver: 192.168.1.1
net0: name=eth0,bridge=vmbr30,firewall=1,hwaddr=96:D2:62:33:2E:39,ip=dhcp,ip6=auto,type=veth
ostype: debian
rootfs: local-lvm:vm-313-disk-0,mountoptions=lazytime;noatime,size=4G
swap: 2048
unprivileged: 1
Click to expand...

Confirmed!
The console is not even showing..
"failed waiting for client: timed out
TASK ERROR: command '/usr/bin/termproxy 5900 --path /vms/101 --perm VM.Console -- /usr/bin/dtach -A /var/run/dtach/vzctlconsole101 -r winch -z lxc-console -n 101 -e -1' failed: exit code 1 "
 
Last edited:
I've tested creating a new LXC CT on an Intel machine running PVE 7.1-8. It's been upgraded to latest Proxmox no-subscription release, but has not been reboot (Not sure if reboot is a safe option at this stage). Running 5.11.22 kernel.

Interestingly the Intel machine is not affected by this issue. My AMD machine is affected.
 
Managed to fix the issue. Now my CTs are once again starting normally.

In my /etc/fstab I customised the /proc filesystem with the following line:

Code: 
proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=1 0 0

By reverting the line to Proxmox's default, then reboot, the problem is solved:

Code: 
proc /proc proc defaults 0 0

Phew!
 
  • Like
Reactions: Stoiko Ivanov and Dunuin
vmware said:
Brand new created unprivileged container is not working, created from official debian 11.0 template:
Click to expand...
tried that here - without any modifications - and cannot reproduce the issue.

vmware said:
In my /etc/fstab I customised the /proc filesystem with the following line:
Click to expand...
this seems like the likely reason for the problems - congrats on finding this so fast!
a default container usually does not have any line in /etc/fstab (apart from a comment) - did you modify /proc on your PVE-node?

@nintendo424 - just for confirmation - do you also have custom options in fstab for proc?
 
vmware said:
Managed to fix the issue. Now my CTs are once again starting normally.

In my /etc/fstab I customised the /proc filesystem with the following line:

Code: 
proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=1 0 0

By reverting the line to Proxmox's default, then reboot, the problem is solved:

Code: 
proc /proc proc defaults 0 0

Phew!
Click to expand...

I'm sorry but i didnt understand.
what is the one we need to put in fstab file?

" proc /proc proc defaults,nosuid,nodev,noexec,relatime,hidepid=1 0 0 "
"proc /proc proc defaults 0 0 "
 
Ok ;)
I just formated my machine with Proxmox v7.1-4 to solve all the problems caused by the latest v7.1-8 version.
Currently the server machine is not randomly rebooting again and all LXC and VM's are working with no problems.
 
Stoiko Ivanov said:
tried that here - without any modifications - and cannot reproduce the issue.


this seems like the likely reason for the problems - congrats on finding this so fast!
a default container usually does not have any line in /etc/fstab (apart from a comment) - did you modify /proc on your PVE-node?

@nintendo424 - just for confirmation - do you also have custom options in fstab for proc?
Click to expand...
I do not have custom options in my fstab for /proc, and my LXC containers were also brand new ones based on the Debian template. I will double check my fstab when I get a chance and post back. It does sound like this is the fix though, just not sure what would have changed the fstab entry.
 
Checking my fstab entry, it was in fact different. It said
Bash: 
proc /proc proc defaults,noatime 0 0

I updated it to just have defaults and sure enough, unprivileged containers are fine now.
 
Glad that worked out for you as well!
 
panic :eek:
Same problem: after updating the host, none of the CT start.
Error:
Code: 
safe_mount: 1200 Operation not permitted - Failed to mount "proc" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc"
lxc_mount_auto_mounts: 810 Operation not permitted - Failed to mount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" with flags 14
lxc_setup: 4356 Failed to setup first automatic mounts
do_start: 1274 Failed to setup container "100"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2068 Failed to spawn container "100"
TASK ERROR: startup for container '100' failed

But mount options are default. /etc/fstab :
Code: 
# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0
/rpool/data/subvol-101-disk-0/etc/fstab :
Code: 
# UNCONFIGURED FSTAB FOR BASE SYSTEM

Brand new created unprivileged container is not working, created from official debian 11.0 template

What to do? need help!
thnx

PS. All VMs are working fine.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back