Thin Overprovisioning (out of space on Thin LVM)

[My4tic]

Hi,

One of my VM does not start (hang on "recovering journal"). I free some space on other VM (same disk) but it did not help. The problem is disk free space. How can I fix it?

  LV            VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotzD- 254.50g             100.00 47.37
  root          pve -wi-ao----  92.75g
  swap          pve -wi-ao----   8.00g
  vm-100-disk-1 pve Vwi-aotz-- 110.00g data        71.32
  vm-101-disk-1 pve Vwi-a-tz-- 110.00g data        89.79
  vm-102-disk-1 pve Vwi-aotz--  80.00g data        96.60

 

[My4tic]

root@:~# lvs -v
  LV            VG  #Seg Attr       LSize   Maj Min KMaj KMin Pool Origin Data%  Meta%  Move Cpy%Sync Log Convert LV UUID                                LProfile
  data          pve    1 twi-aotzD- 254.50g  -1  -1  253    4             100.00 47.37                            LKkGgF-yj2F-HHmb-BVlD-2Fsl-xJ2l-1Nm4KD
  root          pve    1 -wi-ao----  92.75g  -1  -1  253    1                                                     gmpU4M-o3xD-KOfr-oX1S-83fH-sutq-edHL8P
  swap          pve    1 -wi-ao----   8.00g  -1  -1  253    0                                                     e5XAzT-B09z-aeo6-EwPl-r54z-drsq-g3Gpuo
  vm-100-disk-1 pve    1 Vwi-aotz-- 110.00g  -1  -1  253    8 data        71.32                                   imvUo4-ieB2-UMDr-OSGR-i1b2-LWPS-f2wzlR
  vm-101-disk-1 pve    1 Vwi-aotz-- 110.00g  -1  -1  253    7 data        89.79                                   JDVoyo-33FJ-cs4w-Jryd-gVQl-570v-GossgU
  vm-102-disk-1 pve    1 Vwi-aotz--  80.00g  -1  -1  253    6 data        96.60                                   sAV9Ha-1UCx-3Yu3-PAsj-KdeO-Zhwg-Kyqr5j
root@:~# vgs -v
  VG  Attr   Ext   #PV #LV #SN VSize   VFree  VG UUID                                VProfile
  pve wz--n- 4.00m   1   6   0 371.25g 15.75g JgA3u6-1xnF-cH7C-WCso-Sge4-b2vK-60ALdv
root@:~# lsblk -a
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0                          7:0    0         0 loop
loop1                          7:1    0         0 loop
loop2                          7:2    0         0 loop
loop3                          7:3    0         0 loop
loop4                          7:4    0         0 loop
loop5                          7:5    0         0 loop
loop6                          7:6    0         0 loop
loop7                          7:7    0         0 loop
sda                            8:0    0 371.5G  0 disk
├─sda1                         8:1    0     1M  0 part
├─sda2                         8:2    0   256M  0 part
└─sda3                         8:3    0 371.3G  0 part
  ├─pve-swap                 253:0    0     8G  0 lvm  [SWAP]
  ├─pve-root                 253:1    0  92.8G  0 lvm  /
  ├─pve-data_tmeta           253:2    0   128M  0 lvm
  │ └─pve-data-tpool         253:4    0 254.5G  0 lvm
  │   ├─pve-data             253:5    0 254.5G  0 lvm
  │   ├─pve-vm--102--disk--1 253:6    0    80G  0 lvm
  │   ├─pve-vm--101--disk--1 253:7    0   110G  0 lvm
  │   └─pve-vm--100--disk--1 253:8    0   110G  0 lvm
  └─pve-data_tdata           253:3    0 254.5G  0 lvm
    └─pve-data-tpool         253:4    0 254.5G  0 lvm
      ├─pve-data             253:5    0 254.5G  0 lvm
      ├─pve-vm--102--disk--1 253:6    0    80G  0 lvm
      ├─pve-vm--101--disk--1 253:7    0   110G  0 lvm
      └─pve-vm--100--disk--1 253:8    0   110G  0 lvm
root@:~# lvscan -v
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [92.75 GiB] inherit
  ACTIVE            '/dev/pve/data' [254.50 GiB] inherit
  ACTIVE            '/dev/pve/vm-102-disk-1' [80.00 GiB] inherit
  ACTIVE            '/dev/pve/vm-101-disk-1' [110.00 GiB] inherit
  ACTIVE            '/dev/pve/vm-100-disk-1' [110.00 GiB] inherit

On VM:

df -h vm-101-disk-1

Filesystem      Size  Used Avail Use% Mounted on
udev            7.7G     0  7.7G   0% /dev
tmpfs           1.6G  8.5M  1.6G   1% /run
/dev/sda1       101G   12G   85G  12% /
tmpfs           7.7G     0  7.7G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.7G     0  7.7G   0% /sys/fs/cgroup
tmpfs           1.6G     0  1.6G   0% /run/user/0

df -h vm-102-disk-1

/dev/mapper/centos-root   10G  1,7G  8,4G  17% /
devtmpfs                 7,8G     0  7,8G   0% /dev
tmpfs                    7,8G     0  7,8G   0% /dev/shm
tmpfs                    7,8G   17M  7,8G   1% /run
tmpfs                    7,8G     0  7,8G   0% /sys/fs/cgroup
/dev/sda1               1014M  256M  759M  26% /boot
/dev/mapper/centos-home   68G  7,7G   61G  12% /home
tmpfs                    1,6G     0  1,6G   0% /run/user/0

 

[oguz]

LV VG #Seg Attr LSize Maj Min KMaj KMin Pool Origin Data% Meta% Move Cpy%Sync Log Convert LV UUID LProfile data pve 1 twi-aotzD- 254.50g -1 -1 253 4 100.00 47.37 LKkGgF-yj2F-HHmb-BVlD-2Fsl-xJ2l-1Nm4KD

Looks like "data" is full?

 

[My4tic]

Yes, I think so. But when I delete some files from other VM (same disk) nothing change.

 

[oguz]

You can try using fstrim on the VMs, or you can create a new storage and move some of your VMs there. Another option is to resize the LV.