Mountpoints for LXC containers broken after update

W

WoetWoet

Active Member
Feb 22, 2020
4
1
43
34
After running apt upgrade and rebooting, my mountpoints on LXC containers behave differently:
1. Regular mountpoints keep getting mounted as root:root, e.g. /var/lib/mysql, preventing MySQL from starting.
2. Read-only mountpoints prevent the LXC container from starting entirely.

Code: 
root@proxmox:/var/log/apt# pveversion -v
proxmox-ve: 9.1.0 (running kernel: 6.17.4-2-pve)
pve-manager: 9.1.5 (running version: 9.1.5/80cf92a64bef6889)
proxmox-kernel-helper: 9.0.4
proxmox-kernel-6.17.4-2-pve-signed: 6.17.4-2
proxmox-kernel-6.17: 6.17.4-2
proxmox-kernel-6.14.11-5-pve-signed: 6.14.11-5
proxmox-kernel-6.14: 6.14.11-5
proxmox-kernel-6.14.11-4-pve-signed: 6.14.11-4
amd64-microcode: 3.20250311.1
ceph-fuse: 19.2.3-pve2
corosync: 3.1.9-pve2
criu: 4.1.1-1
frr-pythontools: 10.4.1-1+pve1
ifupdown2: 3.3.0-1+pmx11
intel-microcode: 3.20251111.1~deb13u1
libjs-extjs: 7.0.0-5
libproxmox-acme-perl: 1.7.0
libproxmox-backup-qemu0: 2.0.2
libproxmox-rs-perl: 0.4.1
libpve-access-control: 9.0.5
libpve-apiclient-perl: 3.4.2
libpve-cluster-api-perl: 9.0.7
libpve-cluster-perl: 9.0.7
libpve-common-perl: 9.1.7
libpve-guest-common-perl: 6.0.2
libpve-http-server-perl: 6.0.5
libpve-network-perl: 1.2.5
libpve-rs-perl: 0.11.4
libpve-storage-perl: 9.1.0
libspice-server1: 0.15.2-1+b1
lvm2: 2.03.31-2+pmx1
lxc-pve: 6.0.5-4
lxcfs: 6.0.4-pve1
novnc-pve: 1.6.0-3
proxmox-backup-client: 4.1.2-1
proxmox-backup-file-restore: 4.1.2-1
proxmox-backup-restore-image: 1.0.0
proxmox-firewall: 1.2.1
proxmox-kernel-helper: 9.0.4
proxmox-mail-forward: 1.0.2
proxmox-mini-journalreader: 1.6
proxmox-offline-mirror-helper: 0.7.3
proxmox-widget-toolkit: 5.1.5
pve-cluster: 9.0.7
pve-container: 6.1.0
pve-docs: 9.1.2
pve-edk2-firmware: not correctly installed
pve-esxi-import-tools: 1.0.1
pve-firewall: 6.0.4
pve-firmware: 3.17-2
pve-ha-manager: 5.1.0
pve-i18n: 3.6.6
pve-qemu-kvm: 10.1.2-5
pve-xtermjs: 5.5.0-3
qemu-server: 9.1.4
smartmontools: 7.4-pve1
spiceterm: 3.4.1
swtpm: 0.8.0+pve3
vncterm: 1.9.1
zfsutils-linux: 2.3.4-pve1
root@proxmox:/var/log/apt#

Regular mount, works fine:
Code: 
root@proxmox:/# pct config 103
arch: amd64
cores: 4
features: nesting=1
hostname: dev
memory: 8192
mp0: /data/subvol-100-disk-6,mp=/home/logs,size=0T
onboot: 1
ostype: ubuntu
protection: 1
rootfs: vms:subvol-103-disk-0,size=60G
startup: order=100,up=180
swap: 4096
unprivileged: 1
lxc.cgroup2.cpuset.cpus: 6,7,10,11
root@proxmox:/# pct status 103
status: stopped
root@proxmox:/# pct start 103
root@proxmox:/# pct stop 103
root@proxmox:/#

Read-only mountpoint, refusing to start:
Code: 
root@proxmox:/# pct config 103
arch: amd64
cores: 4
features: nesting=1
hostname: dev
memory: 8192
mp0: /data/subvol-100-disk-6,mp=/home/logs,ro=1,size=0T
onboot: 1
ostype: ubuntu
protection: 1
rootfs: vms:subvol-103-disk-0,size=60G
startup: order=100,up=180
swap: 4096
unprivileged: 1
lxc.cgroup2.cpuset.cpus: 6,7,10,11
root@proxmox:/# pct status 103
status: stopped
root@proxmox:/# pct start 103
run_buffer: 571 Script exited with status 30
lxc_init: 845 Failed to run lxc.hook.pre-start for container "103"
__lxc_start: 2046 Failed to initialize container "103"
startup for container '103' failed
root@proxmox:/#

Code: 
root@proxmox:/# lxc-start -n 103 -F -lDEBUG -o lxc-103.log
lxc-start: 103: ../src/lxc/utils.c: run_buffer: 571 Script exited with status 30
lxc-start: 103: ../src/lxc/start.c: lxc_init: 845 Failed to run lxc.hook.pre-start for container "103"
lxc-start: 103: ../src/lxc/start.c: __lxc_start: 2046 Failed to initialize container "103"
lxc-start: 103: ../src/lxc/tools/lxc_start.c: lxc_start_main: 307 The container failed to start
lxc-start: 103: ../src/lxc/tools/lxc_start.c: lxc_start_main: 312 Additional information can be obtained by setting the --logfile and --logpriority options
root@proxmox:/# cat lxc-103.log
lxc-start 103 20260204020827.254 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2295 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 103 20260204020827.254 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2295 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 103 20260204020827.255 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 103 20260204020827.255 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "103", config section "lxc"
lxc-start 103 20260204020827.621 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 103 lxc pre-start produced output: failed to propagate uid and gid to mountpoint: Read-only file system

lxc-start 103 20260204020827.632 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 30
lxc-start 103 20260204020827.632 ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "103"
lxc-start 103 20260204020827.632 ERROR    start - ../src/lxc/start.c:__lxc_start:2046 - Failed to initialize container "103"
lxc-start 103 20260204020827.632 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "103", config section "lxc"
lxc-start 103 20260204020828.134 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "103", config section "lxc"
lxc-start 103 20260204020828.483 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp5: not mounted.

lxc-start 103 20260204020828.483 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp5' failed: exit code 32

lxc-start 103 20260204020828.494 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp4: not mounted.

lxc-start 103 20260204020828.494 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32

lxc-start 103 20260204020828.497 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp2: not mounted.

lxc-start 103 20260204020828.497 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp2' failed: exit code 32

lxc-start 103 20260204020828.500 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp3: not mounted.

lxc-start 103 20260204020828.501 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp3' failed: exit code 32

lxc-start 103 20260204020828.504 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp1: not mounted.

lxc-start 103 20260204020828.504 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 103 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32

lxc-start 103 20260204020828.536 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:307 - The container failed to start
lxc-start 103 20260204020828.536 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:312 - Additional information can be obtained by setting the --logfile and --logpriority options
root@proxmox:/#
 
Last edited:
It's recommended to use apt full-upgrade over apt upgrade maybe just doing the full upgrade will resolve your issue. Does this also happen for new containers?
 
Is this going to be fixed or are we stuck at pve-container 6.0.18 now?

From the linked issue:
- Container: Privileged LXC with NFS bind mount via autofs
Click to expand...

In my setup, this affected unprivileged containers. I do not have any privileged LXC containers.
 
Last edited:
unfortunately, I also got caught on this. It's affecting all unprivileged containers with bind mounts. I caught mine by trying to migrate a container after updating.

It doesn't matter if its RO or not. My NFS share is RW. Current workaround is to downgrade "apt install pve-container=6.0.18" on the pve host.

Code: 
run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "1234"
__lxc_start: 2046 Failed to initialize container "1234"
TASK ERROR: startup for container '1234' failed


Also, the easiest way to upgrade is to just use the GUI. It will run all the right commands and clean up for you. otherwise, use the dist-upgrade recommendation. I have inflicted much self pain after accidentally running apt upgrade myself and wondering why the versions for everything were all funky and pve was not behaving as expected.
 
Last edited:
miztahsparklez said:
unfortunately, I also got caught on this. It's affecting all unprivileged containers with bind mounts. I caught mine by trying to migrate a container after updating.

It doesn't matter if its RO or not. My NFS share is RW. Current workaround is to downgrade "apt install pve-container=6.0.18" on the pve host.

Code: 
run_buffer: 571 Script exited with status 1
lxc_init: 845 Failed to run lxc.hook.pre-start for container "1234"
__lxc_start: 2046 Failed to initialize container "1234"
TASK ERROR: startup for container '1234' failed


Also, the easiest way to upgrade is to just use the GUI. It will run all the right commands and clean up for you. otherwise, use the dist-upgrade recommendation. I have inflicted much self pain after accidentally running apt upgrade myself and wondering why the versions for everything were all funky and pve was not behaving as expected.
Click to expand...
I also just got hit and had to revert, hopefully they will fix it in the near future
 
Just wanted to echo that this has stopped me for starting my LXC containers after updating Proxmox today.

Current workaround is to downgrade "apt install pve-container=6.0.18" on the pve host.
Click to expand...

This fixed the problem for me for now. Thank you very much for posting this!
 
Running the apt install pve-container=6.0.18 fixed it for me for now but it's a downgrade of the package.

So what is the official fix to get the newest container package? I've tried apt dist-upgrade which loaded the newest package but it failed to start my containers so had to revert back to older version.

EDIT: I've used the Web GUI to preform the updates this morning so whatever happened it didn't run correctly for the new package to install properly.
 
Last edited:
Maybe this can be fixed by removing the code that's causing this issue in /usr/share/perl5/PVE/LXC.pm:

Code: 
PVE::Tools::fchownat(fileno($mount_fd), '', $uid, $gid, PVE::Tools::AT_EMPTY_PATH)
    or die "failed to propagate uid and gid to mountpoint: $!\n";

Feel free to test. There is no official fix yet.

For now, I have put the package on hold with apt-mark hold pve-container to prevent accidental upgrading. If you do this, do not forget to unhold when a fix is released.
 
  • Like
Reactions: Darkk
You must log in or register to reply here.
Top Bottom
Back