LXC Fails to start after reboot

[ConorGrocock]

I've spent the past couple days stuck on this issue. Before I rebooted it there were some issue with the container not being able to write to the drive. But when I tried with the host I could read and write without issue.

lxc-start -n 100 -F -lDEBUG -o lxc-100.log
lxc-start: 100: conf.c: run_buffer: 323 Script exited with status 255
lxc-start: 100: start.c: lxc_init: 797 Failed to run lxc.hook.pre-start for container "100"
lxc-start: 100: start.c: __lxc_start: 1896 Failed to initialize container "100"
lxc-start: 100: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 100: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options

cat lxc-100.log
lxc-start 100 20210806124718.326 INFO confile - confile.c:set_config_idmaps:2055 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 100 20210806124718.327 INFO     confile - confile.c:set_config_idmaps:2055 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 100 20210806124718.327 INFO     lsm - lsm/lsm.c:lsm_init:29 - LSM security driver AppArmor
lxc-start 100 20210806124718.327 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
lxc-start 100 20210806124720.495 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/mp0: can't read superblock on /dev/loop7.

lxc-start 100 20210806124720.534 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 100 lxc pre-start produced output: command 'mount /dev/loop7 /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 100 20210806124720.555 ERROR    conf - conf.c:run_buffer:323 - Script exited with status 255
lxc-start 100 20210806124720.555 ERROR    start - start.c:lxc_init:797 - Failed to run lxc.hook.pre-start for container "100"
lxc-start 100 20210806124720.555 ERROR    start - start.c:__lxc_start:1896 - Failed to initialize container "100"
lxc-start 100 20210806124720.555 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "100", config section "lxc"
lxc-start 100 20210806124721.597 INFO     conf - conf.c:run_script_argv:340 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "100", config section "lxc"
lxc-start 100 20210806124722.127 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp0: not mounted.

lxc-start 100 20210806124722.127 DEBUG    conf - conf.c:run_buffer:312 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 100 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32

lxc-start 100 20210806124722.176 ERROR    lxc_start - tools/lxc_start.c:main:308 - The container failed to start
lxc-start 100 20210806124722.176 ERROR    lxc_start - tools/lxc_start.c:main:314 - Additional information can be obtained by setting the --logfile and --logpriority options

The drive is accessible and can be read from and written to.

/dev/sda1             2.7T  2.7T     0 100% /mnt/pve/large-storage
/mnt/pve/large-storage/images/100# ls -lh
total 2.7T
-rw-r----- 1 root root 3.0T Aug  6 13:47 vm-100-disk-0.raw

/etc/pve/lxc/100.conf

arch: amd64
cores: 4
features: nesting=1
hostname: media-server
memory: 4096
mp0: large-storage:100/vm-100-disk-0.raw,mp=/media,size=3000G
net0: name=eth0,bridge=vmbr0,gw=192.168.0.1,hwaddr=9A:3B:05:C1:DA:EB,ip=192.168.3.40/16,type=veth
onboot: 1
ostype: ubuntu
rootfs: VM-Storage:100/vm-100-disk-0.raw,size=50G
swap: 2048
unprivileged: 1

 

[Stoiko Ivanov]

could you try to fsck the container?
`pct fsck 100`?

 

[ConorGrocock]

That didn't do the mount point right?

root@proxmox:~# pct fsck 100
fsck from util-linux 2.33.1
/mnt/pve/VM-Storage/images/100/vm-100-disk-0.raw: clean, 156959/3276800 files, 3779498/13107200 blocks

 

[Stoiko Ivanov]

That didn't do the mount point right?

yes - sorry my mistake!

you should be able to run fsck against the image manually:
`fsck /mnt/pve/large-storage/images/100/vm-100-disk-0.raw`
(assuming large-storage is mounted at the default location)

I hope this helps!

 

[ConorGrocock]

From the looks of it. I expanded the mp large than the size of the disk, what's the next step from here?

root@proxmox:~# fsck /mnt/pve/large-storage/images/100/vm-100-disk-0.raw
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
/mnt/pve/large-storage/images/100/vm-100-disk-0.raw: recovering journal
Error writing block 740294660 (No space left on device).  Ignore error<y>?