[SOLVED] CT won't start after node disk is full

[kotakomputer]

One of my CT 6056 (350GB at sdb1) can not save file, after checking I see the CT 6056 hdd usage is 30% but sdb1 was 100%. So I shutdown CT 6056, then move some files from sdb1, now sbd1 has 32% free space.

When I start CT 6056, error occures:

Job for pve-container@6056.service failed because the control process exited with error code.
See "systemctl status pve-container@6056.service" and "journalctl -xe" for details.

TASK ERROR: command 'systemctl start pve-container@6056' failed: exit code 1

root@sds-002:~# systemctl status pve-container@6056.service
● pve-container@6056.service - PVE LXC Container: 6056
   Loaded: loaded (/lib/systemd/system/pve-container@.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2018-10-18 09:55:11 UTC; 5s ago
     Docs: man:lxc-start
           man:lxc
           man:pct
  Process: 2143 ExecStart=/usr/bin/lxc-start -n 6056 (code=exited, status=1/FAILURE)

Oct 18 09:55:10 sds-002 systemd[1]: Starting PVE LXC Container: 6056...
Oct 18 09:55:11 sds-002 systemd[1]: pve-container@6056.service: Control process exited, code=exited
Oct 18 09:55:11 sds-002 systemd[1]: Failed to start PVE LXC Container: 6056.
Oct 18 09:55:11 sds-002 systemd[1]: pve-container@6056.service: Unit entered failed state.
Oct 18 09:55:11 sds-002 systemd[1]: pve-container@6056.service: Failed with result 'exit-code'.

root@sds-002:~#

Any idea to fix it?

 

[wbumiller]

So the image containing the container was on the disk that ran full? It probably fails to mount -you can test that via `pct mount $vmid` as root - if that fails as well, you can try running a `pct fsck $vmid`...
The problem is that most regular file systems, including ext4, really are not designed to cope with an out-of-space error on their underlying storage...

 

[kotakomputer]

/mnt/data/images/6056/vm-6056-disk-1.raw: Unattached inode 21104135

/mnt/data/images/6056/vm-6056-disk-1.raw: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
command 'fsck -a -l /mnt/data/images/6056/vm-6056-disk-1.raw' failed: exit code 4

Thanks very much ... the CT run fine now, only latest uploaded file missing, fortunately I still have that latest file.

You right, I use EXT4 ... is there any File System able to avoid corruption when out-of-space?
Maybe I have to avoid CT having space more than physical space?