lost thin volume

Mark D · Oct 3, 2017

Hi, I have a server on which I lost a thin volume and I cannot figure out how to recover it. After installing 4.4, I followed these directions to add the thin volume on which I restored several VMs. This morning I came in and the server stopped on a reboot -- power outage I'm thinking.

I've tried everthing I can find online: vgchange -a y pve; lvchange -a y pve -- this one gives me a device-mapper error for each image that exists on the thin volume that's missing.

Here's some stats of the machine:

Code:

root@bmppve:~# lvs -v
  LV            VG  #Seg Attr       LSize   Maj Min KMaj KMin Pool Origin Data%  Meta%  Move Cpy%Sync Log Convert LV UUID                                LProfile
  data          pve    1 twi-aotz--   7.16t  -1  -1  253    4             0.00   0.42                             WMs7N0-eV2a-tvqu-t6j2-VJaw-3Dg2-ynVEHN         
  data_meta0    pve    1 -wi-a----- 116.00m  -1  -1  253    6                                                     n1cT0p-0Ny2-wRrv-03wx-dvI7-Ycwy-CU2rKc         
  data_meta1    pve    1 -wi-a----- 116.00m  -1  -1  253    7                                                     66Iwyu-pfHw-fzOQ-ZoQe-FCDf-wlnK-3YRjie         
  root          pve    1 -wi-ao----  96.00g  -1  -1  253    0                                                     B4PQ8O-3R3K-zmQs-MMNe-279N-6fsC-gUVd9v         
  swap          pve    1 -wi-ao----   8.00g  -1  -1  253    1                                                     D8gcRG-8jxh-VlLd-fyow-nmU0-bacK-InbDNJ         
  vm-102-disk-1 pve    1 Vwi---tz-- 800.00g  -1  -1   -1   -1 data                                                NeemGt-PQuf-n3oZ-iR9U-Jxr9-GuKh-g9xpgw         
  vm-108-disk-1 pve    1 Vwi---tz-- 140.00g  -1  -1   -1   -1 data                                                ZuYcvL-QIzM-lRIl-IVad-uMad-0FUs-D1eb92         
  vm-202-disk-1 pve    1 Vwi---tz--   4.88t  -1  -1   -1   -1 data                                                EfBoC1-6yID-4Oo8-J3z2-NLJY-HrVL-aRqcUk         
  vm-310-disk-1 pve    1 Vwi---tz--  32.00g  -1  -1   -1   -1 data                                                ZmzpD8-yZ8X-6V3T-ZWlK-JWl2-qhQB-GcjrRj         
  vm-500-disk-1 pve    1 Vwi---tz-- 350.00g  -1  -1   -1   -1 data                                                AE8l1l-4ijc-Mc1R-eaRm-S5fj-AtZN-V0XB2x         
  vz            pve    1 Vwi---tz--   2.00t  -1  -1   -1   -1 data                                                gAAXZX-CBM5-7Mm9-zPR3-LT6W-sqdw-hUPkTf         
root@bmppve:~# vgs -v
  VG  Attr   Ext   #PV #LV #SN VSize VFree  VG UUID                                VProfile
  pve wz--n- 4.00m   1  11   0 7.28t 15.66g yiyWmu-ahYD-sIZ9-M2fW-kJQd-H0bM-OnYNVZ         
root@bmppve:~# lsblk -a
NAME                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                  7:0    0        0 loop
loop1                  7:1    0        0 loop
loop2                  7:2    0        0 loop
loop3                  7:3    0        0 loop
loop4                  7:4    0        0 loop
loop5                  7:5    0        0 loop
loop6                  7:6    0        0 loop
loop7                  7:7    0        0 loop
sda                    8:0    0  7.3T  0 disk
├─sda1                 8:1    0    1M  0 part
├─sda2                 8:2    0  256M  0 part
└─sda3                 8:3    0  7.3T  0 part
  ├─pve-root         253:0    0   96G  0 lvm  /
  ├─pve-swap         253:1    0    8G  0 lvm  [SWAP]
  ├─pve-data_tmeta   253:2    0  116M  0 lvm 
  │ └─pve-data-tpool 253:4    0  7.2T  0 lvm 
  │   └─pve-data     253:5    0  7.2T  0 lvm 
  ├─pve-data_tdata   253:3    0  7.2T  0 lvm 
  │ └─pve-data-tpool 253:4    0  7.2T  0 lvm 
  │   └─pve-data     253:5    0  7.2T  0 lvm 
  ├─pve-data_meta0   253:6    0  116M  0 lvm 
  └─pve-data_meta1   253:7    0  116M  0 lvm 
root@bmppve:~# lvscan -v
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  ACTIVE            '/dev/pve/data' [7.16 TiB] inherit
  inactive          '/dev/pve/vm-108-disk-1' [140.00 GiB] inherit
  inactive          '/dev/pve/vz' [2.00 TiB] inherit
  inactive          '/dev/pve/vm-202-disk-1' [4.88 TiB] inherit
  inactive          '/dev/pve/vm-310-disk-1' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-500-disk-1' [350.00 GiB] inherit
  inactive          '/dev/pve/vm-102-disk-1' [800.00 GiB] inherit
  ACTIVE            '/dev/pve/data_meta0' [116.00 MiB] inherit
  ACTIVE            '/dev/pve/data_meta1' [116.00 MiB] inherit
root@bmppve:~# vgscan -v 
    Wiping cache of LVM-capable devices
    Wiping internal VG cache
  Reading all physical volumes.  This may take a while...
  Found volume group "pve" using metadata type lvm2

root@bmppve:~# lvchange -a y pve
  device-mapper: reload ioctl on (253:8) failed: No data available
  device-mapper: reload ioctl on (253:8) failed: No data available
  device-mapper: reload ioctl on (253:8) failed: No data available
  device-mapper: reload ioctl on (253:8) failed: No data available
  device-mapper: reload ioctl on (253:8) failed: No data available
  device-mapper: reload ioctl on (253:8) failed: No data available

then running this:
lvconvert --repair pve/vz
Using default stripesize 64.00 KiB.
Operation not permitted on thin LV pve/vz.
Operations permitted on a thin LV are:
--merge

Any suggestions? I have full backups only 1 day old, so I'm not worried about the data, but I'm worried why this happened. I don't want to have to deal with this again. too much down-time.

udo · Oct 4, 2017

Hi Mark,

lvconvert --repair pve/vz

pve/vz can't be the thin-volume because it's 2TB in size and your vm-202-disk are allready 4TB!
And the meta-data belongs to data.

Does this work?

Code:

lvconvert --repair pve/data

Udo

Mark D · Oct 4, 2017

No the correct storage I want to retrieve is the pve/vz. This is the first time I'm noticing that the disk for VM 202 is at 5TB. I'm sure I had intended to do it as a 500 GB. The backup for that lxc container is 2.33GB so it's not close to reaching the 2TB that the volume had.

this is the result of that command you asked for:

Code:

root@bmppve:~# lvconvert --repair pve/data
  Using default stripesize 64.00 KiB.
  WARNING: Sum of all thin volume sizes (8.17 TiB) exceeds the size of thin pools and the size of whole volume group (7.28 TiB)!
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Only inactive pool can be repaired.
root@bmppve:~# lvchange -a n pve/data
root@bmppve:~# lvconvert --repair pve/data
  Using default stripesize 64.00 KiB.
  WARNING: recovery of pools without pool metadata spare LV is not automated.
  WARNING: If everything works, remove pve/data_meta2 volume.
  WARNING: Use pvmove command to move pve/data_tmeta on the best fitting PV.

What you explained brings me to a question. According to the directions here: https://pve.proxmox.com/wiki/LVM2 - It suggests to mount the newly created LV-thin volume in /var/lib/vz - which is what I did. With that the lxc containers are now also mounted inside that storage, right?

Thanks

udo · Oct 4, 2017

Hi,
you have overcommit your thin-VG! 8.17 TiB used space instead of 7.16TiB real space - that smells like dataloss.

I don't have experiences how solve this. I assume you need space (additional PV, or remove an lv?!).

Udo

Mark D · Oct 5, 2017

I'm hoping someone would be able to help. I'm really not worried about the data. I just want to understand why this happened. From my understanding, LV-thin volumes are made so you can over-provision. Actual usage should never exceed or else you'll have data corruption. In this case there is no chance that the usage exceeded the total space available. My Total backup of all VM's was 351GB.

jeffwadsworth · Oct 5, 2017

https://en.wikipedia.org/wiki/Thin_provisioning

charlesg · Jul 30, 2018

Mark, I know this was over 2 years ago but did you ever figure out what happened? I'm not sure overcommitting was your problem.

I just ran into something very similar. My metapool filled and I started getting all kinds of error. I was able to extend it and reboot but it seems pve/data, the thin provisioning volume had issues.

I ran (like you) 'lvconvert --repair pve/data' which seem to work but I lost ALL my thin provisioned volumes.

charlesg · Aug 18, 2018

It turns out filing up the metadatapool and running `lvconvert --repair` corrupted everything. I used the thin-provisioning-tools (https://github.com/jthornber/thin-provisioning-tools) and some advice from knowledgeable folks to recover.

I documented the process here:

https://www.unixrealm.com/?p=12000

Search

Search

lost thin volume

Mark D

New Member

udo

Distinguished Member

Mark D

New Member

udo

Distinguished Member

Mark D

New Member

jeffwadsworth

Member

charlesg

New Member

charlesg

New Member