PVE8: LXC containers on lvm storage fail to start after upgrade with acl=0

D

dlasher

Renowned Member
Mar 23, 2011
247
36
93
Container fails to start after upgrade, not seeing anything obvious.

lxc-start 61001 20230704010253.978 DEBUG conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--61001--disk--0, missing codepage or helper program, or other error.

mounting it by hand works, so I don't think the LVM data is damaged.

mount /dev/mapper/pve-vm-61001--disk--0 /tmp/1

I was able to rsync data out of the app folders for backup, no read errors. Location also looks valid to me:

lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data -> ../dm-5
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data_tdata -> ../dm-3
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data_tmeta -> ../dm-2
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-data-tpool -> ../dm-4
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-root -> ../dm-1
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-swap -> ../dm-0
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-vm--61001--disk--0 -> ../dm-6
lrwxrwxrwx 1 root root 7 Jul 3 13:26 pve-vm--6106--disk--0 -> ../dm-7


debug is less than helpful.

Code: 
lxc-start 61001 20230704010251.828 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010251.828 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010251.829 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 61001 20230704010251.829 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010253.978 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--61001--disk--0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

lxc-start 61001 20230704010253.979 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: command 'mount -o lazytime,noatime,nosuid,noacl /dev/dm-6 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 61001 20230704010254.109 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 32
lxc-start 61001 20230704010254.110 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "61001"
lxc-start 61001 20230704010254.110 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "61001"
lxc-start 61001 20230704010254.112 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010254.517 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010256.577 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: umount: /var/lib/lxc/61001/rootfs: not mounted

lxc-start 61001 20230704010256.578 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/61001/rootfs' failed: exit code 1

lxc-start 61001 20230704010256.610 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 61001 20230704010256.610 ERROR    start - ../src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "61001"
lxc-start 61001 20230704010256.610 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 61001 20230704010256.610 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start 61001 20230704010913.285 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010913.286 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 61001 20230704010913.287 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 61001 20230704010913.287 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010915.558 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--61001--disk--0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

lxc-start 61001 20230704010915.558 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 61001 lxc pre-start produced output: command 'mount -o lazytime,noatime,nosuid,noacl /dev/dm-6 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 61001 20230704010915.592 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 32
lxc-start 61001 20230704010915.592 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "61001"
lxc-start 61001 20230704010915.592 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "61001"
lxc-start 61001 20230704010915.593 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010916.100 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "61001", config section "lxc"
lxc-start 61001 20230704010918.180 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: umount: /var/lib/lxc/61001/rootfs: not mounted

lxc-start 61001 20230704010918.181 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 61001 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/61001/rootfs' failed: exit code 1

lxc-start 61001 20230704010918.216 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
lxc-start 61001 20230704010918.216 ERROR    start - ../src/lxc/start.c:lxc_end:985 - Failed to run lxc.hook.post-stop for container "61001"
lxc-start 61001 20230704010918.216 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 61001 20230704010918.216 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

And proxmox CAN mount it correctly, because I can "move storage" from one storage to another, and it's rsync'ing as needed

root 96165 1.0 0.1 12556 9208 ? SN 18:10 0:11 rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system --bwlimit=0 /var/lib/lxc/61001/.copy-volume-2/ /var/lib/lxc/61001/.copy-volume-1
root 96166 0.5 0.1 156012 12272 ? SN 18:10 0:05 rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system --bwlimit=0 /var/lib/lxc/61001/.copy-volume-2/ /var/lib/lxc/61001/.copy-volume-1
root 96168 4.6 0.1 296732 12464 ? DN 18:10 0:50 rsync --stats -X -A --numeric-ids -aH --whole-file --sparse --one-file-system --bwlimit=0 /var/lib/lxc/61001/.copy-volume-2/ /var/lib/lxc/61001/.copy-volume-1


Only two LXC containers on this host, other one started just fine.
 
Last edited:
Update: Moved to ZFS storage - VM starts fine. Moved back to local-LVM - fails to start, same error message.

setting "ACL = disabled" in the UI (acl=0 in the conf file) prevents the container from starting. Worked fine under pve7. Works fine on ZFS.


DOES NOT WORK ON LVM:
rootfs: local-lvm:vm-61001-disk-0,acl=0,mountoptions=lazytime;noatime;nosuid,size=20G

WORKS ON LVM:
rootfs: local-lvm:vm-61001-disk-0,size=20G,mountoptions=lazytime;noatime;nosuid

I've searched the release notes, didn't find anything about ACL changes. Is this an intended change?
 
Last edited:
here is what worked for me

Code: 
pct restore 130 my_pbs_server:backup/ct/130/2023-02-11T04:20:52Z --storage data --rootfs data:10,mountoptions=noatime,quota=0
 
Last edited:
Gre
ned said:
here what worked for me

Code: 
pct restore 130 my_pbs_server:backup/ct/130/2023-02-11T04:20:52Z --storage data --rootfs data:10,mountoptions=noatime,quota=0
Click to expand...
Great suggestion - hope this gets fixed in 8.1?
 
You must log in or register to reply here.
Top Bottom