LXC won't boot after resize

[spanaghi]

I resized my LXC using the web interface from 500G to 700G. I then restarted the container, but it won't load. When I run pct start 100 I get the following error:

root@pve:~# pct start 100
run_buffer: 571 Script exited with status 32
lxc_init: 845 Failed to run lxc.hook.pre-start for container "100"
__lxc_start: 2034 Failed to initialize container "100"
startup for container '100' failed

I then ran e2fsck to see what's going on, and I get the following error. No matter what I do/say y or no to, it doesn't help. I also tried to pull the backup superblock, but that doesn't help either:

root@pve:~# e2fsck -fy /dev/pve/vm-100-disk-0
e2fsck 1.47.0 (5-Feb-2023)
The filesystem size (according to the superblock) is 235929600 blocks
The physical size of the device is 183500800 blocks
Either the superblock or the partition table is likely to be corrupt!
Abort? yes

I can't mount the PVE either. Any help to get this to work would be amazing.

Other files just in case they are needed​

Config​

root@pve:~# pct config 100
arch: amd64
cores: 10
features: fuse=1,nesting=1
hostname: Debian
memory: 32000
mp0: local-lvm:vm-100-disk-3,mp=/mnt/test,backup=1,size=8G
mp1: /mnt/pve/nextcloud,mp=/mnt/nextcloud
mp22: /mnt/pve/immich_backup,mp=/data/compose/78/backup
mp3: /mnt/pve/movies,mp=/mnt/movies
mp4: /mnt/pve/tv,mp=/mnt/tv
mp5: /mnt/pve/download,mp=/mnt/download
mp7: /mnt/pve/tv,mp=/mnt/tv
mp8: /mnt/pve/backup,mp=/mnt/backup
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.4.1,hwaddr=BC:24:11:91:8B:77,ip=192.168.6.250/24,ip6=dhcp,type=veth
onboot: 1
ostype: debian
rootfs: local-lvm:vm-100-disk-0,size=700G
startup: order=1
swap: 4096
unprivileged: 1
unused0: immich:100/vm-100-disk-0.raw
lxc.cgroup.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

lxc-100.log​

root@pve:~# cat lxc-100.log 
lxc-start 100 20240720212422.312 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20240720212422.313 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20240720212422.313 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 100 20240720212422.313 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20240720212422.727 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--100--disk--0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
lxc-start 100 20240720212422.727 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: command 'mount /dev/dm-6 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32
lxc-start 100 20240720212422.739 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 32
lxc-start 100 20240720212422.739 ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20240720212422.739 ERROR    start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "100"
lxc-start 100 20240720212422.739 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20240720212423.241 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20240720212423.532 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/100/rootfs: not mounted
lxc-start 100 20240720212423.532 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/100/rootfs' failed: exit code 1
lxc-start 100 20240720212423.543 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
lxc-start 100 20240720212423.543 ERROR    start - ../src/lxc/start.c:lxc_end:986 - Failed to run lxc.hook.post-stop for container "100"
lxc-start 100 20240720212423.543 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:307 - The container failed to start
lxc-start 100 20240720212423.543 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:312 - Additional information can be obtained by setting the --logfile and --logpriority options
lxc-start 100 20240722142144.714 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20240722142144.714 INFO     confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20240722142144.714 INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 100 20240722142144.714 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20240722142145.787 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/mapper/pve-vm--100--disk--0, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
lxc-start 100 20240722142145.789 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: command 'mount /dev/dm-6 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32
lxc-start 100 20240722142145.892 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 32
lxc-start 100 20240722142145.892 ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20240722142145.892 ERROR    start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "100"
lxc-start 100 20240722142145.893 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20240722142145.591 INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20240722142145.872 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/100/rootfs: not mounted
lxc-start 100 20240722142145.872 DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount --recursive -- /var/lib/lxc/100/rootfs' failed: exit code 1
lxc-start 100 20240722142145.881 ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
lxc-start 100 20240722142145.882 ERROR    start - ../src/lxc/start.c:lxc_end:986 - Failed to run lxc.hook.post-stop for container "100"
lxc-start 100 20240722142145.882 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:307 - The container failed to start
lxc-start 100 20240722142145.882 ERROR    lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:312 - Additional information can be obtained by setting the --logfile and --logpriority options

 

[Moayad]

Hi,

Thank you for the syslog!

I hope you have a backup of your CT.

Could you check the `vm-100-disk-0` size using `lvdisplay`? In your case, the command should look like the following:

lvdisplay /dev/pve/vm-100-disk-0

Could you also check of the storage update disk size using `pct rescan`:

pct rescan --vmid 100

 

[spanaghi]

Hi Moayad,

Thanks for getting back to me. I ended up getting it to work by adjusting the size of the container a bit at a time. I honestly don't understand how it fixed it, but it finally did. Posted below are the results of the two commands you asked for:

root@pve:~# lvdisplay /dev/pve/vm-100-disk-0
  --- Logical volume ---
  LV Path                /dev/pve/vm-100-disk-0
  LV Name                vm-100-disk-0
  VG Name                pve
  LV UUID                XbsOLZ-azcH-rlKf-tKaw-3KGq-BYXD-UC6IFn
  LV Write Access        read/write
  LV Creation host, time pve, 2024-03-06 15:40:09 -0500
  LV Pool name           data
  WARNING: LV pve/vm-100-disk-0 maps <494.29 GiB while the size is only 494.00 GiB.
  LV Status              available
  # open                 1
  LV Size                494.00 GiB
  Mapped size            100.00%
  Current LE             126464
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           252:6

root@pve:~# pct rescan --vmid 100
rescan volumes...
  WARNING: Thin volume pve/vm-100-disk-0 maps 530736873472 while the size is only 530428461056.
  WARNING: Thin volume pve/vm-100-disk-0 maps 530736873472 while the size is only 530428461056.
Use of uninitialized value in hash element at /usr/share/perl5/PVE/LXC.pm line 2228.
Use of uninitialized value in hash element at /usr/share/perl5/PVE/LXC.pm line 2228.
CT 100: add unreferenced volume 'local-lvm:vm-100-disk-2' as 'unused2' to config.
CT 100: add unreferenced volume 'local-lvm:vm-100-disk-1' as 'unused3' to config.
CT 100: updated volume size of '/mnt/pve/nextcloud' in config.
CT 100: updated volume size of '/mnt/pve/docmost' in config.
CT 100: updated volume size of '/mnt/pve/movies' in config.
CT 100: updated volume size of '/mnt/pve/tv' in config.
CT 100: updated volume size of '/mnt/pve/download' in config.
CT 100: updated volume size of '/mnt/pve/library' in config.
CT 100: updated volume size of '/mnt/pve/recipes' in config.
CT 100: updated volume size of '/mnt/pve/backup' in config.
CT 100: updated volume size of '/mnt/pve/wavelog' in config.