Data recovery from LVM

millido

New Member
Sep 7, 2021
1
0
1
24
I have a lvm where the metadata got full, I didn't understand what was wrong and attempted to fix it with

Code:
lvconvert --repair pve/data

This turned out to be a bad idea, as it now says that there is no data for any lv's. Similar to this question https://unix.stackexchange.com/ques...over-lvm-thin-pool-volume-after-failed-repair, but I never tried to extend metadata, and I cannot get that solution to work.

After the research I have done, it seems like i can restore the metadata from archive. This is a ProxMox install, and so all things are vm-drives. There is a archive file for "before running lvconvert --repair". There is also a file "before lvcreate" for a vmdisk, which I don't care about. And I know that the LVM worked for a few months after I created that disk. So if it is possible to save the rest of the data using that, that is perfectly acceptable. At this point I am trying to mitigate dataloss more than get the pool running as it should.

So, I have cloned the disk to a new (larger) disk using dd to try to be safe here. And I have attempted to restore from archive using a live image with this command

Code:
lvchange -an pvm
pvcreate --uuid "z34DFR-Kkk6-4P5m-N1uy-n7dh-sI11-ABuxCl" --restorefile /etc/lvm/archive/pve_00064-1664281480.vg /dev/sda

But that gives

Code:
Couldn't find device with uuid z34DFR-Kkk6-4P5m-N1uy-n7dh-sI11-ABuxCl.
UUID z34DFR-Kkk6-4P5m-N1uy-n7dh-sI11-ABuxCl already in use on "/dev/sda3".

Here are some info on the LVM and the disks. /dev/sdb is currently a clone of /dev/sda.
Code:
root@frank:~# fdisk -l
Disk /dev/sda: 1.8 TiB, 1999844147200 bytes, 3905945600 sectors
Disk model: PERC 6/i        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: EAE945C0-902E-4C35-8544-F5D0BE412BF3

Device       Start        End    Sectors  Size Type
/dev/sda1       34       2047       2014 1007K BIOS boot
/dev/sda2     2048    1050623    1048576  512M EFI System
/dev/sda3  1050624 3905945566 3904894943  1.8T Linux LVM


Disk /dev/sdb: 5.5 TiB, 5999532441600 bytes, 11717836800 sectors
Disk model: PERC 6/i        
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 1D53AC81-30BC-BF46-BEA9-B61C0FF235DF

Device     Start        End    Sectors  Size Type
/dev/sdb1   2048 5859377151 5859375104  2.7T Linux filesystem


Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-root: 128 GiB, 137438953472 bytes, 268435456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-data_meta0: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Code:
root@frank:~# pvs -a
  PV         VG  Fmt  Attr PSize  PFree  
  /dev/sda2           ---      0       0 
  /dev/sda3  pve lvm2 a--  <1.82t 211.00g
  /dev/sdb1           ---      0       0 

root@frank:~# vgs -a
  VG  #PV #LV #SN Attr   VSize  VFree  
  pve   1  21   0 wz--n- <1.82t 211.00g
    
root@frank:~# lvs -a
  LV              VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve twi-aotz--  <1.44t             0.00   0.15                            
  data_meta0      pve -wi-a-----  15.00g                                                    
  [data_tdata]    pve Twi-ao----  <1.44t                                                    
  [data_tmeta]    pve ewi-ao----  15.00g                                                    
  [lvol1_pmspare] pve ewi-------  15.00g                                                    
  root            pve -wi-ao---- 128.00g                                                    
  swap            pve -wi-ao----   8.00g                                                    
  vm-100-disk-0   pve Vwi---tz--   8.00g data                                               
  vm-201-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-205-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-300-disk-0   pve Vwi---tz--  24.00g data                                               
  vm-301-disk-0   pve Vwi---tz--  32.00g data                                               
  vm-302-disk-0   pve Vwi---tz--  64.00g data                                               
  vm-400-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-401-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-402-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-403-disk-0   pve Vwi---tz-- 128.00g data                                               
  vm-404-disk-0   pve Vwi---tz-- 256.00g data                                               
  vm-506-disk-0   pve Vwi---tz--  32.00g data                                               
  vm-508-disk-0   pve Vwi---tz-- 800.00g data                                               
  vm-508-disk-1   pve Vwi---tz--  24.00g data                                               
  vm-600-disk-0   pve Vwi---tz--  16.00g data                                               
  vm-601-disk-0   pve Vwi---tz--  16.00g data                                               
  vm-602-disk-0   pve Vwi---tz--  32.00g data
Code:
root@frank:~# pvscan
  PV /dev/sda3   VG pve             lvm2 [<1.82 TiB / 211.00 GiB free]
  Total: 1 [<1.82 TiB] / in use: 1 [<1.82 TiB] / in no VG: 0 [0   ]

root@frank:~# lvscan
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [128.00 GiB] inherit
  ACTIVE            '/dev/pve/data' [<1.44 TiB] inherit
  inactive          '/dev/pve/vm-100-disk-0' [8.00 GiB] inherit
  inactive          '/dev/pve/vm-201-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-600-disk-0' [16.00 GiB] inherit
  inactive          '/dev/pve/vm-601-disk-0' [16.00 GiB] inherit
  inactive          '/dev/pve/vm-506-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-602-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-205-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-300-disk-0' [24.00 GiB] inherit
  inactive          '/dev/pve/vm-508-disk-0' [800.00 GiB] inherit
  inactive          '/dev/pve/vm-508-disk-1' [24.00 GiB] inherit
  inactive          '/dev/pve/vm-400-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-401-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-402-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-403-disk-0' [128.00 GiB] inherit
  inactive          '/dev/pve/vm-301-disk-0' [32.00 GiB] inherit
  inactive          '/dev/pve/vm-302-disk-0' [64.00 GiB] inherit
  inactive          '/dev/pve/vm-404-disk-0' [256.00 GiB] inherit
  ACTIVE            '/dev/pve/data_meta0' [15.00 GiB] inherit

root@frank:~# vgdisplay 
  --- Volume group ---
  VG Name               pve
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  125
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                21
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <1.82 TiB
  PE Size               4.00 MiB
  Total PE              476671
  Alloc PE / Size       422655 / 1.61 TiB
  Free  PE / Size       54016 / 211.00 GiB
  VG UUID               H0G0xv-ddq1-BdvI-fIxF-dg8e-BT2X-CqYWFX

Thank you so much for any answers or tips on this. As I mentioned, data recovery is starting to become first priority. If I am able to provide any more things to make it easier to give tips I will!

Also, if I try to lvchange -ay pve, I get this for all the vm-disks. device-mapper: reload ioctl on (253: 10) failed: No data available.
 
I have a similar situation...and there are many others on here with similar stories. But none with a single reply.

I am just fumbling. Thought snapshots sounded cool and decided to try windows 10vm on lvm knowing nothing about it (just assumed it was another brand of ext that permitted snapshots). I didn't know the drive would fill up exponentially fast at each reboot having not checked discard. Heck of a learning experience. :)

Did you get it figured out and at least recover the data?
 
Same same.. waking up to a VM reboot during multiple power failures (outage during weather), VMs wont start - tried the
Code:
lvconvert --repair pve/data
since it was mentioned in the many first posts given the error message.

Now i'm here with a 0.0% data usage, i have the *.vg's already saved. It seems the pve-data_meta0 is corrupted during the repair command.



Basically my lvscan looks really like @millido's:


Code:
root@charly:/etc/lvm/archive# lvscan -a
  ACTIVE            '/dev/pve/data' [<1.67 TiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  inactive          '/dev/pve/vm-100-disk-1' [500.00 GiB] inherit
  inactive          '/dev/pve/vm-101-disk-0' [50.00 GiB] inherit
  inactive          '/dev/pve/vm-100-disk-2' [10.00 GiB] inherit
  inactive          '/dev/pve/vm-104-disk-0' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-init_with_ssh' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_init_with_ssh' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-wp_init_install' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_wp_init_install' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-WebsiteFinished' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_WebsiteFinished' [200.00 GiB] inherit
  inactive          '/dev/pve/vm-104-state-Website_06062024' [<16.49 GiB] inherit
  inactive          '/dev/pve/snap_vm-104-disk-0_Website_06062024' [200.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240908120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240908120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240908180002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240908180002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240909060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240909060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240909120002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240909120002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240909180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240909180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240910060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240910060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240910120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240910120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240910180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240910180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240911060002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240911060002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240911120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240911120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240911180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240911180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240912060002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240912060002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240912120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240912120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240912180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240912180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240913060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240913060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240913120002' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240913120002' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240913180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240913180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240913180003' [10.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240914060003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240914060003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240914060003' [10.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240914120003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240914120003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240914120003' [10.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-0_autodaily240914180003' [800.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-1_autodaily240914180003' [500.00 GiB] inherit
  inactive          '/dev/pve/snap_vm-100-disk-2_autodaily240914180003' [10.00 GiB] inherit
  ACTIVE            '/dev/pve/data_meta0' [<15.88 GiB] inherit
  ACTIVE            '/dev/pve/data_tdata' [<1.67 TiB] inherit
  ACTIVE            '/dev/pve/data_tmeta' [<15.88 GiB] inherit
  inactive          '/dev/pve/lvol1_pmspare' [<15.88 GiB] inherit

Code:
root@charly:/etc/lvm/archive# lvchange -ay pve
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available
  device-mapper: reload ioctl on  (252:7) failed: No data available

Does anyone have the same issue and maybe can help me recovering/restoring the meta0 file? I also found some topics about thin-provisioning-tools but no luck so far.
 
Those 15.88gb metadata partitions are a hard limit meaning your metadata is likely corrupted. There are LVM tools, but I was unable to repair mine.

Recovery software was able to get my files, but the data I was storing was not common and was recovered in an unusable format (most are made to find docs and pictures ).

I ended up wiping the system and putting a robust backup system in place to make it take approximately 2hrs to rebuild the entire project.

Painful learning experience. And sorry I cannot offer hope
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!