Cant start LXC after expanding disk

[godsavethequ33n]

I've messed something up along the way and need a little guidance to see where I have gone wrong!

Kernel Version Linux 6.5.11-7-pve (2023-12-05T09:44Z)
Manager Version pve-manager/8.1.3/b46aac3b42da5d15

Spoiler: pct start error from log

lxc-start 109 20240228182716.757 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 109 20240228182716.757 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "109", config section "lxc"
lxc-start 109 20240228182717.309 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 109 lxc pre-start produced output: failed to get device path

lxc-start 109 20240228182717.326 ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 2
lxc-start 109 20240228182717.327 ERROR    start - ../src/lxc/start.c:lxc_init:844 - Failed to run lxc.hook.pre-start for container "109"
lxc-start 109 20240228182717.327 ERROR    start - ../src/lxc/start.c:__lxc_start:2027 - Failed to initialize container "109"
lxc-start 109 20240228182717.327 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "109", config section "lxc"
lxc-start 109 20240228182717.876 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp1: not mounted.

lxc-start 109 20240228182717.876 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp1' failed: exit code 32

lxc-start 109 20240228182717.883 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp2: not mounted.

lxc-start 109 20240228182717.884 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp2' failed: exit code 32

lxc-start 109 20240228182717.889 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp4: not mounted.

lxc-start 109 20240228182717.889 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp4' failed: exit code 32

lxc-start 109 20240228182717.908 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp3: not mounted.

lxc-start 109 20240228182717.908 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp3' failed: exit code 32

lxc-start 109 20240228182717.916 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp0: not mounted.

lxc-start 109 20240228182717.916 DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 109 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 109 20240228182717.968 INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "109", config section "lxc"
lxc-start 109 20240228182718.471 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:306 - The container failed to start
lxc-start 109 20240228182718.471 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

Spoiler: pct config

arch: amd64
cores: 2
description:removed
features: nesting=1
hostname: storj
memory: 4096
mp0: ahsnap:vm-109-disk-0,mp=/data/storj,size=16T
net0: name=eth0,bridge=vmbr0,hwaddr=96:44:F4:81:19:1F,ip=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-109-disk-0,size=20G
swap: 512
tags: storj
lxc.cgroup2.devices.allow: a
lxc.cap.drop:

Spoiler: GUI for "ahsnap" storage

Spoiler: journalctl

Feb 28 14:26:06 pm pct[105879]: <root@pam> starting task UPID:pm:00019DE8:0005EF8B:65DF88CE:vzstart:109:root@pam:
Feb 28 14:26:06 pm pct[105960]: starting CT 109: UPID:pm:00019DE8:0005EF8B:65DF88CE:vzstart:109:root@pam:
Feb 28 14:26:07 pm systemd[1]: Started pve-container@109.service - PVE LXC Container: 109.
Feb 28 14:26:08 pm kernel: EXT4-fs (dm-14): mounted filesystem caf8b13d-f585-4529-999d-bb79de589cbc r/w with ordered data mode. Quota mode: none.
Feb 28 14:26:08 pm kernel: EXT4-fs warning (device dm-3): ext4_multi_mount_protect:318: fsck is running on the filesystem
Feb 28 14:26:08 pm kernel: EXT4-fs warning (device dm-3): ext4_multi_mount_protect:318: MMP failure info: last update time: 1709143478, last update node: pm, last update device: /dev/ahsnap/vm-109-disk-0
Feb 28 14:26:08 pm pct[105960]: startup for container '109' failed
Feb 28 14:26:08 pm pct[105879]: <root@pam> end task UPID:pm:00019DE8:0005EF8B:65DF88CE:vzstart:109:root@pam: startup for container '109' failed
Feb 28 14:26:08 pm kernel: EXT4-fs (dm-14): unmounting filesystem caf8b13d-f585-4529-999d-bb79de589cbc.
Feb 28 14:26:09 pm systemd[1]: pve-container@109.service: Main process exited, code=exited, status=1/FAILURE
Feb 28 14:26:09 pm systemd[1]: pve-container@109.service: Failed with result 'exit-code'.
Feb 28 14:26:09 pm systemd[1]: pve-container@109.service: Consumed 1.194s CPU time.

Interesting it references dm3 which doesn't seem to exist in /dev/

Alos /dev/ahsnap/vm-109-disk-0 doesnt seem to exist when I check via CLI???

Spoiler: storage.cfg

lvm: ahsnap
        vgname ahsnap
        content rootdir,images
        nodes pm
        shared 0

So it appears it is running a fschk on the disk.. which is why I cannot start the LXC is that correct? When I check the process list I dont see anything running:

root@pm:~# ps aux | grep fsck
root      114859  0.0  0.0   6332  2304 pts/0    S+   14:34   0:00 grep fsck
root@pm:~#

Spoiler: misc

root@pm:/dev# lvs
  LV            VG       Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-109-disk-0 ahsnap   -wi-------   16.00t

root@pm:/dev# vgs
  VG       #PV #LV #SN Attr   VSize    VFree
  ahsnap     1   1   0 wz--n-   16.37t <380.00g

root@pm:/dev# pvs
  PV             VG       Fmt  Attr PSize    PFree
  /dev/sdae      ahsnap   lvm2 a--    16.37t <380.00g


root@pm:/dev/mapper# ls -la
lrwxrwxrwx  1 root root       8 Feb 28 13:21 pve-vm--109--disk--0 -> ../dm-14


lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sdae                          65:224  0  16.4T  0 disk

lsblk -f
NAME                         FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINTS
sdae                         LVM2_member LVM2 001       JthxDB-nV6Z-j3wY-s8rA-iszN-bdEb-qNqVHw


vgscan
  Found volume group "ahsnap" using metadata type lvm2

lvscan
  inactive          '/dev/ahsnap/vm-109-disk-0' [16.00 TiB] inherit

pvesm status
Name                Type     Status           Total            Used       Available        %
ahsnap               lvm     active     17578323968     17179869184       398454784   97.73%

Another thing I see - the size of the disk is reflected at 17.59 TB in the GUI when I check the CT Volumes tab. Under the containers resource tab it shows the mountpoint at 16TB. I believe I messed something up there as well?




Thank you for helping me learn!

 

[godsavethequ33n]

Its almost as if the mount point ahsnap doesnt exist any more?

 

[godsavethequ33n]

lvdisplay
  --- Logical volume ---
  LV Path                /dev/ahsnap/vm-109-disk-0
  LV Name                vm-109-disk-0
  VG Name                ahsnap
  LV UUID                NvdDf0-MLVf-kaMc-p9zd-lS3J-2Dj3-a13LV9
  LV Write Access        read/write
  LV Creation host, time pm, 2022-05-28 22:19:39 -0400
  LV Status              NOT available
  LV Size                16.00 TiB
  Current LE             4194304
  Segments               6
  Allocation             inherit
  Read ahead sectors     auto

It shows /dev/ahsnap but /dev/ahsnap no longer exists..

 

[godsavethequ33n]

Partially solved:
vgchange -ay
pct fsck 109 -device mp0

Produced error:
MMP check failed: If you are sure the filesystem is not in use on any node, run:
'tune2fs -f -E clear_mmp /dev/mapper/ahsnap-vm--109--disk--0'

Ran requested command and fsck again. Container started without error.

There is still the discrepancy of the actual disk space that is being reported mismatching as show in the original post. Any suggestions there would be appreciated greatly!!