Hard reboot, now no way to boot LXC containers

[past12]

Hi all,
during a backup something seemed to have gone wrong: the whole machine got stuck and I had to hard reboot the server.

When booting up again, one of my lxc containers (the one I was backing up) is unable to boot.

Just tried to `lxc-start -n 101 -F -lDEBUG -o lxc-101.log` and the output is as follows:

lxc-start 101 20221130155109.874 INFO     confile - confile.c:set_config_idmaps:2112 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 101 20221130155109.874 INFO     confile - confile.c:set_config_idmaps:2112 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 101 20221130155109.875 INFO     lsm - lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 101 20221130155109.875 INFO     conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "101", config section "lxc"
lxc-start 101 20221130155301.675 DEBUG    conf - conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 101 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--101--disk--0, missing codepage or helper program, or other error.

lxc-start 101 20221130155301.675 DEBUG    conf - conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 101 lxc pre-start produced output: command 'mount /dev/dm-8 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

lxc-start 101 20221130155301.687 ERROR    conf - conf.c:run_buffer:321 - Script exited with status 32
lxc-start 101 20221130155301.687 ERROR    start - start.c:lxc_init:847 - Failed to run lxc.hook.pre-start for container "101"
lxc-start 101 20221130155301.687 ERROR    start - start.c:__lxc_start:2008 - Failed to initialize container "101"
lxc-start 101 20221130155301.687 INFO     conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "101", config section "lxc"
lxc-start 101 20221130155302.189 INFO     conf - conf.c:run_script_argv:337 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "101", config section "lxc"
lxc-start 101 20221130155302.739 DEBUG    conf - conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 101 lxc post-stop produced output: umount: /var/lib/lxc/101/rootfs: not mounted

lxc-start 101 20221130155302.740 DEBUG    conf - conf.c:run_buffer:310 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 101 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/101/rootfs' failed: exit code 1

lxc-start 101 20221130155302.751 ERROR    conf - conf.c:run_buffer:321 - Script exited with status 1
lxc-start 101 20221130155302.751 ERROR    start - start.c:lxc_end:988 - Failed to run lxc.hook.post-stop for container "101"
lxc-start 101 20221130155302.751 ERROR    lxc_start - tools/lxc_start.c:main:306 - The container failed to start
lxc-start 101 20221130155302.751 ERROR    lxc_start - tools/lxc_start.c:main:311 - Additional information can be obtained by setting the --logfile and --logpriority options

Tried to `pct fsck 101` as suggested on the forum, but I'm stuck at:

root@pve:~# pct fsck 101
fsck from util-linux 2.36.1
MMP interval is 27 seconds and total wait time is 110 seconds. Please wait...
/dev/mapper/pve-vm--101--disk--0:
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

fsck.ext4: MMP: device currently active while trying to open /dev/mapper/pve-vm--101--disk--0
command 'fsck -a -l /dev/pve/vm-101-disk-0' failed: exit code 8

lvs -a says:

root@pve:~# lvs -a
  LV                        VG  Attr       LSize    Pool Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  data                      pve twi-aotz-- <141.43g                    44.29  2.56
  [data_tdata]              pve Twi-ao---- <141.43g
  [data_tmeta]              pve ewi-ao----   <1.45g
  [lvol0_pmspare]           pve ewi-------   <1.45g
  root                      pve -wi-ao----   55.75g
  snap_vm-101-disk-0_vzdump pve Vri---tz-k   32.00g data vm-101-disk-0
  swap                      pve -wi-ao----    7.00g
  vm-100-disk-0             pve Vwi-aotz--    4.00m data               0.00
  vm-100-disk-1             pve Vwi-aotz--   32.00g data               97.29
  vm-101-disk-0             pve Vwi-a-tz--   32.00g data               74.39
  vm-102-disk-0             pve Vwi-a-tz--    8.00g data               95.90

Please help

EDIT: I have the very same issue starting another lxc container (102) as well.

 

[past12]

One more bit: trying to create a new lxc container, I get the following message:

WARNING: Thin pool pve/data has unexpected transaction id 14, expecting 13.
TASK ERROR: unable to create CT 103 - lvcreate 'pve/vm-103-disk-0' error:   Cannot create new thin volume, free space in thin pool pve/data reached threshold.