Data recovery from LVM

millido

New Member
Sep 7, 2021
1
0
1
24
I have a lvm where the metadata got full, I didn't understand what was wrong and attempted to fix it with

Code:
lvconvert --repair pve/data

This turned out to be a bad idea, as it now says that there is no data for any lv's. Similar to this question https://unix.stackexchange.com/ques...over-lvm-thin-pool-volume-after-failed-repair, but I never tried to extend metadata, and I cannot get that solution to work.

After the research I have done, it seems like i can restore the metadata from archive. This is a ProxMox install, and so all things are vm-drives. There is a archive file for "before running lvconvert --repair". There is also a file "before lvcreate" for a vmdisk, which I don't care about. And I know that the LVM worked for a few months after I created that disk. So if it is possible to save the rest of the data using that, that is perfectly acceptable. At this point I am trying to mitigate dataloss more than get the pool running as it should.

So, I have cloned the disk to a new (larger) disk using dd to try to be safe here. And I have attempted to restore from archive using a live image with this command

Code:
lvchange -an pvm
pvcreate --uuid "z34DFR-Kkk6-4P5m-N1uy-n7dh-sI11-ABuxCl" --restorefile /etc/lvm/archive/pve_00064-1664281480.vg /dev/sda

But that gives

Code:
Couldn't find device with uuid z34DFR-Kkk6-4P5m-N1uy-n7dh-sI11-ABuxCl.
UUID z34DFR-Kkk6-4P5m-N1uy-n7dh-sI11-ABuxCl already in use on "/dev/sda3".

Here are some info on the LVM and the disks. /dev/sdb is currently a clone of /dev/sda.
Code:
root@frank:~# fdisk -l
Disk /dev/sda: 1.8 TiB, 1999844147200 bytes, 3905945600 sectors
Disk model: PERC 6/i        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EAE945C0-902E-4C35-8544-F5D0BE412BF3

Device       Start        End    Sectors  Size Type
/dev/sda1       34       2047       2014 1007K BIOS boot
/dev/sda2     2048    1050623    1048576  512M EFI System
/dev/sda3  1050624 3905945566 3904894943  1.8T Linux LVM


Disk /dev/sdb: 5.5 TiB, 5999532441600 bytes, 11717836800 sectors
Disk model: PERC 6/i        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1D53AC81-30BC-BF46-BEA9-B61C0FF235DF

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 5859377151 5859375104  2.7T Linux filesystem


Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-root: 128 GiB, 137438953472 bytes, 268435456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-data_meta0: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Code:
root@frank:~# pvs -a
  PV         VG  Fmt  Attr PSize  PFree  
  /dev/sda2           ---      0       0 
  /dev/sda3  pve lvm2 a--  <1.82t 211.00g
  /dev/sdb1           ---      0       0 

root@frank:~# vgs -a
  VG  #PV #LV #SN Attr   VSize  VFree  
  pve   1  21   0 wz--n- <1.82t 211.00g
    
root@frank:~# lvs -a
  LV              VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve twi-aotz--  <1.44t             0.00   0.15                            
  data_meta0      pve -wi-a-----  15.00g                                                    
  [data_tdata]    pve Twi-ao----  <1.44t                                                    
  [data_tmeta]    pve ewi-ao----  15.00g                                                    
  [lvol1_pmspare] pve ewi-------  15.00g                                                    
  root            pve -wi-ao---- 128.00g                                                    
  swap            pve -wi-ao----   8.00g                                                    
  vm-100-disk-0   pve Vwi---tz--   8.00g data                                               
  vm-201-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-205-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-300-disk-0   pve Vwi---tz--  24.00g data                                               
  vm-301-disk-0   pve Vwi---tz--  32.00g data                                               
  vm-302-disk-0   pve Vwi---tz--  64.00g data                                               
  vm-400-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-401-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-402-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-403-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-404-disk-0   pve Vwi---tz-- 256.00g data                                               
  vm-506-disk-0   pve Vwi---tz--  32.00g data                                               
  vm-508-disk-0   pve Vwi---tz-- 800.00g data                                               
  vm-508-disk-1   pve Vwi---tz--  24.00g data                                               
  vm-600-disk-0   pve Vwi---tz--  16.00g data                                               
  vm-601-disk-0   pve Vwi---tz--  16.00g data                                               
  vm-602-disk-0   pve Vwi---tz--  32.00g data
Code:
root@frank:~# pvscan
  PV /dev/sda3   VG pve             lvm2 [<1.82 TiB / 211.00 GiB free]
  Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0   ]

root@frank:~# lvscan
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [128.00 GiB] inherit
  ACTIVE            '/dev/pve/data' [<1.44 TiB] inherit
  inactive          '/dev/pve/vm-100-disk-0' [8.00 GiB] inherit
  inactive          '/dev/pve/vm-201-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-600-disk-0' [16.00 GiB] inherit
  inactive          '/dev/pve/vm-601-disk-0' [16.00 GiB] inherit
  inactive          '/dev/pve/vm-506-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-602-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-205-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-300-disk-0' [24.00 GiB] inherit
  inactive          '/dev/pve/vm-508-disk-0' [800.00 GiB] inherit
  inactive          '/dev/pve/vm-508-disk-1' [24.00 GiB] inherit
  inactive          '/dev/pve/vm-400-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-401-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-402-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-403-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-301-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-302-disk-0' [64.00 GiB] inherit
  inactive          '/dev/pve/vm-404-disk-0' [256.00 GiB] inherit
  ACTIVE            '/dev/pve/data_meta0' [15.00 GiB] inherit

root@frank:~# vgdisplay 
  --- Volume group ---
  VG Name               pve
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  125
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                21
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <1.82 TiB
  PE Size               4.00 MiB
  Total PE              476671
  Alloc PE / Size       422655 / 1.61 TiB
  Free  PE / Size       54016 / 211.00 GiB
  VG UUID               H0G0xv-ddq1-BdvI-fIxF-dg8e-BT2X-CqYWFX

Thank you so much for any answers or tips on this. As I mentioned, data recovery is starting to become first priority. If I am able to provide any more things to make it easier to give tips I will!

Also, if I try to lvchange -ay pve, I get this for all the vm-disks. device-mapper: reload ioctl on (253: 10) failed: No data available.
 
I have a similar situation...and there are many others on here with similar stories. But none with a single reply.

I am just fumbling. Thought snapshots sounded cool and decided to try windows 10vm on lvm knowing nothing about it (just assumed it was another brand of ext that permitted snapshots). I didn't know the drive would fill up exponentially fast at each reboot having not checked discard. Heck of a learning experience. :)

Did you get it figured out and at least recover the data?
 
Same same.. waking up to a VM reboot during multiple power failures (outage during weather), VMs wont start - tried the
Code:
lvconvert --repair pve/data
since it was mentioned in the many first posts given the error message.

Now i'm here with a 0.0% data usage, i have the *.vg's already saved. It seems the pve-data_meta0 is corrupted during the repair command.



Basically my lvscan looks really like @millido's:


Code:
root@charly:/etc/lvm/archive# lvscan -a
  ACTIVE            '/dev/pve/data' [<1.67 TiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  inactive          '/dev/pve/vm-100-disk-1' [500.00 GiB] inherit
  inactive          '/dev/pve/vm-101-disk-0' [50.00 GiB] inherit
  inactive          '/dev/pve/vm-100-disk-2' [10.00 GiB] inherit
  inactive          '/dev/pve/vm-104-disk-0' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-init_with_ssh' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_init_with_ssh' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-wp_init_install' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_wp_init_install' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-WebsiteFinished' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_WebsiteFinished' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-Website_06062024' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_Website_06062024' [200.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240908120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240908120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240908180002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240908180002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240909060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240909060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240909120002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240909120002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240909180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240909180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240910060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240910060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240910120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240910120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240910180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240910180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240911060002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240911060002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240911120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240911120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240911180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240911180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240912060002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240912060002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240912120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240912120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240912180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240912180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240913060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240913060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240913120002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240913120002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240913180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240913180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240913180003' [10.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240914060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240914060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240914060003' [10.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240914120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240914120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240914120003' [10.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240914180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240914180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240914180003' [10.00 GiB] inherit
  ACTIVE            '/dev/pve/data_meta0' [<15.88 GiB] inherit
  ACTIVE            '/dev/pve/data_tdata' [<1.67 TiB] inherit
  ACTIVE            '/dev/pve/data_tmeta' [<15.88 GiB] inherit
  inactive          '/dev/pve/lvol1_pmspare' [<15.88 GiB] inherit

Code:
root@charly:/etc/lvm/archive# lvchange -ay pve
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available

Does anyone have the same issue and maybe can help me recovering/restoring the meta0 file? I also found some topics about thin-provisioning-tools but no luck so far.
 
Those 15.88gb metadata partitions are a hard limit meaning your metadata is likely corrupted. There are LVM tools, but I was unable to repair mine.

Recovery software was able to get my files, but the data I was storing was not common and was recovered in an unusable format (most are made to find docs and pictures ).

I ended up wiping the system and putting a robust backup system in place to make it take approximately 2hrs to rebuild the entire project.

Painful learning experience. And sorry I cannot offer hope