repair thin lv volume

[almiller]

Is there a way to mount the disk on the proxmox server and extend or repair it?
I have pve-manager/7.2-7/d0dd0e85 (running kernel: 5.15.39-3-pve)


I have a vm that boots up but runs out of inodes.
I.e.: I can login and touch 1 file, but touching a 2nd file says "no space left on device"


The VM says inodes are full.
k3s-master:~# mount |grep root cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755,inode64) k3s-master:~# touch file1 k3s-master:~# touch file2 touch: file2: No space left on device k3s-master:~# df -i / Filesystem Inodes Used Available Use% Mounted on /dev/sda3 780288 780287 1 100% / k3s-master:~# df -h / Filesystem Size Used Available Use% Mounted on /dev/sda3 11.6G 1.3G 9.7G 12% /

Here's the LV

root@pve ~ # lvdisplay pve/vm-200-disk-0 --- Logical volume --- LV Path /dev/pve/vm-200-disk-0 LV Name vm-200-disk-0 VG Name pve LV UUID xCAs9R-OdyT-hSby-EtKk-MLA6-JSb1-yuSqId LV Write Access read/write LV Creation host, time pve, 2022-02-19 11:43:27 -0500 LV Pool name data LV Status available # open 0 LV Size 16.00 GiB Mapped size 40.74% Current LE 4096 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 253:19

 

[fiona]

Hi,
if you can still log in, maybe you can also remove some files to free up inodes? To see where the indoes are used, you can use e.g.
du --inodes -hs * | sort -h starting at / and to find candidate directories and then recurse. Of course you should not remove important files, but things like log or cache files.

If not, to mount the filesystem on the host, you can use something like the following (adapted to your volume and partition of course):

root@pve701 ~ # losetup --find --show --partscan /dev/pve/vm-143-disk-0
/dev/loop0
root@pve701 ~ # fdisk -l /dev/loop0
Disk /dev/loop0: 4 GiB, 4294967296 bytes, 8388608 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x5acb29fc

Device       Boot   Start     End Sectors  Size Id Type
/dev/loop0p1 *       2048 7340031 7337984  3.5G 83 Linux
/dev/loop0p2      7342078 8386559 1044482  510M  5 Extended
/dev/loop0p5      7342080 8386559 1044480  510M 82 Linux swap / Solaris
root@pve701 ~ # mount /dev/loop0p1 /mnt/point
root@pve701 ~ # ls /mnt/point
bin   dev  home        initrd.img.old  lib32  libx32      media  opt   root  sbin  sys  usr  vmlinuz
boot  etc  initrd.img  lib             lib64  lost+found  mnt    proc  run   srv   tmp  var  vmlinuz.old

and when you are done freeing up indoes

root@pve701 ~ # umount /mnt/point
root@pve701 ~ # losetup --detach /dev/loop0

 

[almiller]

Thanks fiona, I was able to mount the disk on the PVE server and determined where are the inodes are being used up.

On the VM rootfs had 762.2K inodes

k3s-master:~# df  -hi /
Filesystem              Inodes      Used Available Use% Mounted on
/dev/sda3               762.0K    762.0K         0 100% /

And after mounting the disk on the PVE server I found where the inodes are being used up.

root@pve ~ # losetup --find --show --partscan /dev/pve/vm-200-disk-0
/dev/loop0

root@pve ~ # fdisk -l /dev/loop0
...
/dev/loop0p3      8595456 33554431 24958976 11.9G 83 Linux

root@pve ~ #  du --inodes -hs /mnt/* |sort -hr|head -1
760K    /mnt/var

root@pve ~ # du --inodes -hs /mnt/var/lib/kubelet/pods/e8d1dab0-ca7f-4791-8af6-1cee3e961dc2/containers/* |sort -h
751K    /mnt/var/lib/kubelet/pods/e8d1dab0-ca7f-4791-8af6-1cee3e961dc2/containers/node-exporter

root@pve ~ # find  /mnt/var/lib/kubelet/pods/e8d1dab0-ca7f-4791-8af6-1cee3e961dc2/containers/node-exporter/ -type f -ls |wc -l
768453