LXC container root storage full, not starting

[foggyroses]

Hello!

My LXC container has run out of storage and I want to access the filesystem within the container to delete some larger files, though its locked behind a "vm-102-disk-0.raw" file that I cannot figure out how to access, the container will not start for more than a second at a time, is there any way i can solve this?

 

[foggyroses]

Is there a way to possibly start up a container and mount the raw file to it? Here's the message that shows up every time i start the machine.

run_buffer: 571 Script exited with status 255
lxc_init: 845 Failed to run lxc.hook.pre-start for container "102"
__lxc_start: 2034 Failed to initialize container "102"
TASK ERROR: startup for container '102' failed

Is my data just stuck like this or is there any way of salvaging it?

 

[dcsapak]

you could use 'pct mount' to mount the file system on the host, then delete some files
alternatively you can increase the disk just a bit so that the container is able to start and delete files from within

 

[fiona]

Hi,

My LXC container has run out of storage

how did you determine this?

To get more details about the error during startup, you can use pct start 102 --debug. EDIT: that is already provided here: https://forum.proxmox.com/threads/cant-start-lxc-container.162446/ Please don't double post in the future, it just gets confusing to collect the information.

Since the error is about the filesystem, I'd try running a pct fsck 102 to check/repair the filesystem.

 

[foggyroses]

Hi,

how did you determine this?

To get more details about the error during startup, you can use pct start 102 --debug. EDIT: that is already provided here: https://forum.proxmox.com/threads/cant-start-lxc-container.162446/ Please don't double post in the future, it just gets confusing to collect the information.

Since the error is about the filesystem, I'd try running a pct fsck 102 to check/repair the filesystem.

Sorry about that, i'm quite new to forums in general, i tried the fsck and got the result below

fsck from util-linux 2.38.1
/mnt/pve/gloopertKingston/images/102/vm-102-disk-0.raw:
The superblock could not be read or does not describe a valid ext2/ext3/ext4
filesystem.  If the device is valid and it really contains an ext2/ext3/ext4
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>
 or
    e2fsck -b 32768 <device>

fsck.ext4: Superblock checksum does not match superblock while trying to open /mnt/pve/gloopertKingston/images/102/vm-102-disk-0.raw
command 'fsck -a -l /mnt/pve/gloopertKingston/images/102/vm-102-disk-0.raw' failed: exit code 8

The assumption about it being full came from the containers root disk being completely filled up in the webui.

 

[foggyroses]

you could use 'pct mount' to mount the file system on the host, then delete some files
alternatively you can increase the disk just a bit so that the container is able to start and delete files from within

The container is kept on a single disk that cant really be resized (at least that i know of)

# pct mount 102
mount: /var/lib/lxc/102/rootfs: cannot mount; probably corrupted filesystem on /dev/loop0.

       dmesg(1) may have more information after failed mount system call.

mounting container failed

command 'mount /dev/loop0 /var/lib/lxc/102/rootfs//' failed: exit code 32

Trying to mount the system comes up with a corrupted /dev/loop0

EDIT: that is already provided here

just for clarity heres the result from pct start 102 --debug

# pct start 102 --debug
run_buffer: 571 Script exited with status 255
lxc_init: 845 Failed to run lxc.hook.pre-start for container "102"
__lxc_start: 2034 Failed to initialize container "102"
0 hostid 100000 range 65536
INFO     lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "102", config section "lxc"
DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 102 lxc pre-start produced output: mount: /var/lib/lxc/.pve-staged-mounts/rootfs: cannot mount; probably corrupted filesystem on /dev/loop0.
dmesg(1) may have more information after failed mount system call.

DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 102 lxc pre-start produced output: command 'mount /dev/loop0 /var/lib/lxc/.pve-staged-mounts/rootfs' failed: exit code 32

ERROR    utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 255
ERROR    start - ../src/lxc/start.c:lxc_init:845 - Failed to run lxc.hook.pre-start for container "102"
ERROR    start - ../src/lxc/start.c:__lxc_start:2034 - Failed to initialize container "102"
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "102", config section "lxc"
startup for container '102' failed

 

[fiona]

What does

fsck.ext4 -v /mnt/pve/gloopertKingston/images/102/vm-102-disk-0.raw

say? Please also check the system logs/journal on the host from around the time the original issue occurred for any additional information.

 

[foggyroses]

What does

fsck.ext4 -v /mnt/pve/gloopertKingston/images/102/vm-102-disk-0.raw

say? Please also check the system logs/journal on the host from around the time the original issue occurred for any additional information.

When i tried running this it gave me a lot of errors saying no space left on device, I ran it again and it completely fix the issue completely, i am now able to pct mount and delete the larger files within the server. Thank you so much for your time.