Mountpoints for LXC containers broken after update

W

WoetWoet

Active Member
Feb 22, 2020
4
1
43
34
After running apt upgrade and rebooting, my mountpoints on LXC containers behave differently:
1. Regular mountpoints keep getting mounted as root:root, e.g. /var/lib/mysql, preventing MySQL from starting.
2. Read-only mountpoints prevent the LXC container from starting entirely.

Code: 
root@proxmox:/var/log/apt# pveversion -v
proxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)
pve-manager: 9.1.5 (running version: 9.1.5/80cf92a64bef6889)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.17: 6.17.4-2
proxmox-kernel-6.14.11-5-pve-signed: 6.14.11-5
proxmox-kernel-6.14: 6.14.11-5
proxmox-kernel-6.14.11-4-pve-signed: 6.14.11-4
amd64-microcode: 3.20250311.1
ceph-fuse: 19.2.3-pve2
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx11
intel-microcode: 3.20251111.1~deb13u1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.5
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.0.7
libpve-cluster-perl: 9.0.7
libpve-common-perl: 9.1.7
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.5
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.0
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-4
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.2-1
proxmox-backup-file-restore: 4.1.2-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.5
pve-cluster: 9.0.7
pve-container: 6.1.0
pve-docs: 9.1.2
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.17-2
pve-ha-manager: 5.1.0
pve-i18n: 3.6.6
pve-qemu-kvm: 10.1.2-5
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.4
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1
root@proxmox:/var/log/apt#

Regular mount, works fine:
Code: 
root@proxmox:/# pct config 103
arch: amd64
cores: 4
features: nesting=1
hostname: dev
memory: 8192
mp0: /data/subvol-100-disk-6,mp=/home/logs,size=0T
onboot: 1
ostype: ubuntu
protection: 1
rootfs: vms:subvol-103-disk-0,size=60G
startup: order=100,up=180
swap: 4096
unprivileged: 1
lxc.cgroup2.cpuset.cpus: 6,7,10,11
root@proxmox:/# pct status 103
status: stopped
root@proxmox:/# pct start 103
root@proxmox:/# pct stop 103
root@proxmox:/#

Read-only mountpoint, refusing to start:
Code: 
root@proxmox:/# pct config 103
arch: amd64
cores: 4
features: nesting=1
hostname: dev
memory: 8192
mp0: /data/subvol-100-disk-6,mp=/home/logs,ro=1,size=0T
onboot: 1
ostype: ubuntu
protection: 1
rootfs: vms:subvol-103-disk-0,size=60G
startup: order=100,up=180
swap: 4096
unprivileged: 1
lxc.cgroup2.cpuset.cpus: 6,7,10,11
root@proxmox:/# pct status 103
status: stopped
root@proxmox:/# pct start 103
run_buffer: 571 Script exited with status 30
lxc_init: 845 Failed to run lxc.hook.pre-start for container "103"
__lxc_start: 2046 Failed to initialize container "103"
startup for container '103' failed
root@proxmox:/#

Code: 
root@proxmox:/# lxc-start -n 103 -F -lDEBUG -o lxc-103.log
lxc-start: 103: ../src/lxc/utils.c: run_buffer: 571 Script exited with status 30
lxc-start: 103: ../src/lxc/start.c: lxc_init: 845 Failed to run lxc.hook.pre-start for container "103"
lxc-start: 103: ../src/lxc/start.c: __lxc_start: 2046 Failed to initialize container "103"
lxc-start: 103: ../src/lxc/tools/lxc_start.c: lxc_start_main: 307 The container failed to start
lxc-start: 103: ../src/lxc/tools/lxc_start.c: lxc_start_main: 312 Additional information can be obtained by setting the --logfile and --logpriority options
root@proxmox:/# cat lxc-103.log
lxc-start 103 20260204020827.254 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2295 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 103 20260204020827.254 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2295 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 103 20260204020827.255 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 103 20260204020827.255 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "103", config section "lxc"
lxc-start 103 20260204020827.621 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 103 lxc pre-start produced output: failed to propagate uid and gid to mountpoint: Read-only file system

lxc-start 103 20260204020827.632 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 30
lxc-start 103 20260204020827.632 ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "103"
lxc-start 103 20260204020827.632 ERROR    start - ../src/lxc/start.c:__lxc_start:2046 - Failed to initialize container "103"
lxc-start 103 20260204020827.632 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "103", config section "lxc"
lxc-start 103 20260204020828.134 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "103", config section "lxc"
lxc-start 103 20260204020828.483 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp5: not mounted.

lxc-start 103 20260204020828.483 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp5' failed: exit code 32

lxc-start 103 20260204020828.494 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp4: not mounted.

lxc-start 103 20260204020828.494 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32

lxc-start 103 20260204020828.497 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp2: not mounted.

lxc-start 103 20260204020828.497 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp2' failed: exit code 32

lxc-start 103 20260204020828.500 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp3: not mounted.

lxc-start 103 20260204020828.501 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp3' failed: exit code 32

lxc-start 103 20260204020828.504 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp1: not mounted.

lxc-start 103 20260204020828.504 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32

lxc-start 103 20260204020828.536 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:307 - The container failed to start
lxc-start 103 20260204020828.536 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:312 - Additional information can be obtained by setting the --logfile and --logpriority options
root@proxmox:/#
 
Last edited:
It's recommended to use apt full-upgrade over apt upgrade maybe just doing the full upgrade will resolve your issue. Does this also happen for new containers?
 
Is this going to be fixed or are we stuck at pve-container 6.0.18 now?

From the linked issue:
- Container: Privileged LXC with NFS bind mount via autofs
Click to expand...

In my setup, this affected unprivileged containers. I do not have any privileged LXC containers.
 
Last edited:
unfortunately, I also got caught on this. It's affecting all unprivileged containers with bind mounts. I caught mine by trying to migrate a container after updating.

It doesn't matter if its RO or not. My NFS share is RW. Current workaround is to downgrade "apt install pve-container=6.0.18" on the pve host.

Code: 
run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "1234"
__lxc_start: 2046 Failed to initialize container "1234"
TASK ERROR: startup for container '1234' failed


Also, the easiest way to upgrade is to just use the GUI. It will run all the right commands and clean up for you. otherwise, use the dist-upgrade recommendation. I have inflicted much self pain after accidentally running apt upgrade myself and wondering why the versions for everything were all funky and pve was not behaving as expected.
 
Last edited:
miztahsparklez said:
unfortunately, I also got caught on this. It's affecting all unprivileged containers with bind mounts. I caught mine by trying to migrate a container after updating.

It doesn't matter if its RO or not. My NFS share is RW. Current workaround is to downgrade "apt install pve-container=6.0.18" on the pve host.

Code: 
run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "1234"
__lxc_start: 2046 Failed to initialize container "1234"
TASK ERROR: startup for container '1234' failed


Also, the easiest way to upgrade is to just use the GUI. It will run all the right commands and clean up for you. otherwise, use the dist-upgrade recommendation. I have inflicted much self pain after accidentally running apt upgrade myself and wondering why the versions for everything were all funky and pve was not behaving as expected.
Click to expand...
I also just got hit and had to revert, hopefully they will fix it in the near future
 
You must log in or register to reply here.
Top Bottom
Back