PVE8: LXC containers on lvm storage fail to start after upgrade with acl=0

D

dlasher

Renowned Member
Mar 23, 2011
242
30
93
Container fails to start after upgrade, not seeing anything obvious.

lxc-start 61001 20230704010253.978 DEBUG conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--61001--disk--0, missing codepage or helper program, or other error.

mounting it by hand works, so I don't think the LVM data is damaged.

mount /dev/mapper/pve-vm-61001--disk--0 /tmp/1

I was able to rsync data out of the app folders for backup, no read errors. Location also looks valid to me:

lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data -> ../dm-5
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data_tdata -> ../dm-3
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data_tmeta -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data-tpool -> ../dm-4
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-root -> ../dm-1
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-swap -> ../dm-0
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-vm--61001--disk--0 -> ../dm-6
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-vm--6106--disk--0 -> ../dm-7


debug is less than helpful.

Code: 
lxc-start 61001 20230704010251.828 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010251.828 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010251.829 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 61001 20230704010251.829 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010253.978 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--61001--disk--0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

lxc-start 61001 20230704010253.979 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: command 'mount -o lazytime,noatime,nosuid,noacl /dev/dm-6 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 61001 20230704010254.109 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 32
lxc-start 61001 20230704010254.110 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "61001"
lxc-start 61001 20230704010254.110 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "61001"
lxc-start 61001 20230704010254.112 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010254.517 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010256.577 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: umount: /var/lib/lxc/61001/rootfs: not mounted

lxc-start 61001 20230704010256.578 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/61001/rootfs' failed: exit code 1

lxc-start 61001 20230704010256.610 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 61001 20230704010256.610 ERROR    start - ../src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "61001"
lxc-start 61001 20230704010256.610 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 61001 20230704010256.610 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start 61001 20230704010913.285 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010913.286 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010913.287 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 61001 20230704010913.287 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010915.558 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--61001--disk--0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

lxc-start 61001 20230704010915.558 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: command 'mount -o lazytime,noatime,nosuid,noacl /dev/dm-6 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 61001 20230704010915.592 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 32
lxc-start 61001 20230704010915.592 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "61001"
lxc-start 61001 20230704010915.592 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "61001"
lxc-start 61001 20230704010915.593 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010916.100 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010918.180 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: umount: /var/lib/lxc/61001/rootfs: not mounted

lxc-start 61001 20230704010918.181 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/61001/rootfs' failed: exit code 1

lxc-start 61001 20230704010918.216 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 61001 20230704010918.216 ERROR    start - ../src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "61001"
lxc-start 61001 20230704010918.216 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 61001 20230704010918.216 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

And proxmox CAN mount it correctly, because I can "move storage" from one storage to another, and it's rsync'ing as needed

root 96165 1.0 0.1 12556 9208 ? SN 18:10 0:11 rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system --bwlimit=0 /var/lib/lxc/61001/.copy-volume-2/ /var/lib/lxc/61001/.copy-volume-1
root 96166 0.5 0.1 156012 12272 ? SN 18:10 0:05 rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system --bwlimit=0 /var/lib/lxc/61001/.copy-volume-2/ /var/lib/lxc/61001/.copy-volume-1
root 96168 4.6 0.1 296732 12464 ? DN 18:10 0:50 rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system --bwlimit=0 /var/lib/lxc/61001/.copy-volume-2/ /var/lib/lxc/61001/.copy-volume-1


Only two LXC containers on this host, other one started just fine.
 
Last edited:
Update: Moved to ZFS storage - VM starts fine. Moved back to local-LVM - fails to start, same error message.

setting "ACL = disabled" in the UI (acl=0 in the conf file) prevents the container from starting. Worked fine under pve7. Works fine on ZFS.


DOES NOT WORK ON LVM:
rootfs: local-lvm:vm-61001-disk-0,acl=0,mountoptions=lazytime;noatime;nosuid,size=20G

WORKS ON LVM:
rootfs: local-lvm:vm-61001-disk-0,size=20G,mountoptions=lazytime;noatime;nosuid

I've searched the release notes, didn't find anything about ACL changes. Is this an intended change?
 
Last edited:
here is what worked for me

Code: 
pct restore 130 my_pbs_server:backup/ct/130/2023-02-11T04:20:52Z --storage data --rootfs data:10,mountoptions=noatime,quota=0
 
Last edited:
Gre
ned said:
here what worked for me

Code: 
pct restore 130 my_pbs_server:backup/ct/130/2023-02-11T04:20:52Z --storage data --rootfs data:10,mountoptions=noatime,quota=0
Click to expand...
Great suggestion - hope this gets fixed in 8.1?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom