Need help Lvmthin volume will not load

B

Bobsmyuncle

New Member
Jun 30, 2024
23
2
3
*****EDIT*****
This started as a cry for help. As I'm going through the learning process I'm just documenting progress and links I found. Maybe someday it will help someone.

***The closest I got to help on any of the forums I posted was "Make backups", and most just went unanswered (like this one). This is apparently an area where very few have had success in recovering.

I have found ReclaiMe software (reclaime.com) is able to see the underlying VM files in the LVM. It did not recognize the drive as an LVM drive (probably due to the corrupted metadata), but it looks like I can get my data...at a price (waiting to see what the software wants to charge). This may be the lifeline someone needs if they are desperate like I was.

*****Original post below this line*****

Lvm thin fails on boot.
Error code 1

I believe it is because I didn't select discard when I set up the VM and the metadata is full...also my newbie floundering.

Edit, after freaking out, settling down and going through what happened...I think the last thing I did was resize the LVM. Is there a location in a text file I can access to change the VM size larger maybe? (Maybe still freaking out a little, but working through the stages of grief now ).

Normally it wouldn't be an issue, but I moved to a larger SSD that was my backup drive a few days ago and hadn't made a new backup yet (id10t error). I took a couple of snapshots and those are useless as they are in the container that won't load.

I've searched everywhere I can think of and have tried adding an extra disk through command line. I can add it to VG, but it won't extend lv because lv won't load (somehow it sees it with vgs command though).

If there's anything that may help it would be greatly appreciated.

Last login: Sat Jun 29 15:09:06 PDT 2024 on pts/0
root@proxmox:~# vgs
WARNING: Couldn't find device with uuid miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T.
WARNING: VG pve is missing PV miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T (last written to /dev/sdc).
VG #PV #LV #SN Attr VSize VFree
Daddy_Game_2tb 2 9 0 wz--n- 2.04t <186.68g
pve 2 3 0 wz-pn- <1023.67g 202.31g
root@proxmox:~# lvextend /dev/Daddy_Game_2tb/data -L +150.00g
Logical volume data not found in volume group Daddy_Game_2tb.
root@proxmox:~#
 
Last edited:
I found this while I was waiting for approval...it might be the answer. But will Gparted be able to access the LVM?

https://forum.proxmox.com/threads/decrease-a-vm-disk-size.122430/

https://unix.stackexchange.com/questions/675858/logical-volume-not-found-in-volume-group
I'm going to try this when I get home. It may just be another simple noob mistake I am learning from. Need to check that /data is the correct name as I used it based on PVE/data and may have changed it to another name in the GUI.

Code: 
root@proxmox:~# lsblk -a
NAME                                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0                                   7:0    0     0B  0 loop
loop1                                   7:1    0     0B  0 loop
loop2                                   7:2    0     0B  0 loop
loop3                                   7:3    0     0B  0 loop
loop4                                   7:4    0     0B  0 loop
loop5                                   7:5    0     0B  0 loop
loop6                                   7:6    0     0B  0 loop
loop7                                   7:7    0     0B  0 loop
sda                                     8:0    0 838.4G  0 disk
├─sda1                                  8:1    0  1007K  0 part
├─sda2                                  8:2    0     1G  0 part /boot/efi
└─sda3                                  8:3    0 837.4G  0 part
  ├─pve-swap                          252:2    0     8G  0 lvm  [SWAP]
  ├─pve-root                          252:3    0    96G  0 lvm  /
  ├─pve-data_tmeta                    252:4    0   7.2G  0 lvm
  │ └─pve-data                        252:6    0   703G  0 lvm
  └─pve-data_tdata                    252:5    0   703G  0 lvm
    └─pve-data                        252:6    0   703G  0 lvm
sdb                                     8:16   0 953.9G  0 disk
└─sdb1                                  8:17   0 953.9G  0 part /mnt/pve/Backup
sdc                                     8:32   0 186.3G  0 disk
sdd                                     8:48   0 186.3G  0 disk
sde                                     8:64   0   1.9T  0 disk
├─Daddy_Game_2tb-Daddy_Game_2tb_tmeta 252:0    0  15.9G  0 lvm
└─Daddy_Game_2tb-Daddy_Game_2tb_tdata 252:1    0   1.8T  0 lvm
sr0                                    11:0    1  1024M  0 rom

Code: 
root@proxmox:~# lvs -a
  WARNING: Couldn't find device with uuid miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T.
  WARNING: VG pve is missing PV miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T (last written to /dev/sdc).
  LV                           VG             Attr       LSize    Pool           Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  Daddy_Game_2tb               Daddy_Game_2tb twi---tz--    1.83t                                                                
  [Daddy_Game_2tb_tdata]       Daddy_Game_2tb Twi-a-----    1.83t                                                                
  [Daddy_Game_2tb_tmeta]       Daddy_Game_2tb ewi-a-----  <15.88g                                                                
  [lvol0_pmspare]              Daddy_Game_2tb ewi-------  <15.88g                                                                
  snap_vm-100-disk-0_New2TB    Daddy_Game_2tb Vri---tz-k    1.12t Daddy_Game_2tb                                                  
  snap_vm-100-disk-0_memb4memu Daddy_Game_2tb Vri---tz-k   <2.69t Daddy_Game_2tb                                                  
  snap_vm-100-disk-0_withAsuka Daddy_Game_2tb Vri---tz-k   <2.69t Daddy_Game_2tb                                                  
  snap_vm-100-disk-1_New2TB    Daddy_Game_2tb Vri---tz-k    4.00m Daddy_Game_2tb vm-100-disk-1                                    
  snap_vm-100-disk-1_memb4memu Daddy_Game_2tb Vri---tz-k    4.00m Daddy_Game_2tb vm-100-disk-1                                    
  snap_vm-100-disk-1_withAsuka Daddy_Game_2tb Vri---tz-k    4.00m Daddy_Game_2tb vm-100-disk-1                                    
  vm-100-disk-1                Daddy_Game_2tb Vwi---tz--    4.00m Daddy_Game_2tb                                                  
  vm-100-state-New2TB          Daddy_Game_2tb Vwi---tz-- <128.49g Daddy_Game_2tb                                                  
  data                         pve            twi-a-tz--  703.01g                              0.00   0.26                        
  [data_tdata]                 pve            Twi-ao----  703.01g                                                                
  [data_tmeta]                 pve            ewi-ao----    7.17g                                                                
  [lvol0_pmspare]              pve            ewi-------    7.17g                                                                
  root                         pve            -wi-ao----   96.00g                                                                
  swap                         pve            -wi-ao----    8.00g

after looking at this I am thinking those 2 snapshots that are 2.6tb might be the problem. the biggest issue is that i cannot force delete a snapshot unless the lv gets loaded...which it will not do. my latest little plan of adding my 100gb hard drive to give space just went out the window. :)
 
Last edited:
got rid of the pesky pve failure. not sure if pve is broken now...but it's one less thing to get an error on.

Code: 
root@proxmox:~# pvs -o+pv_used
  WARNING: Couldn't find device with uuid miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T.
  WARNING: VG pve is missing PV miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T (last written to /dev/sdc).
  PV         VG             Fmt  Attr PSize    PFree    Used 
  /dev/sda3  pve            lvm2 a--  <837.36g   16.00g <821.36g
  /dev/sdd   Daddy_Game_2tb lvm2 a--  <186.31g <186.31g       0
  /dev/sde   Daddy_Game_2tb lvm2 a--     1.86t  376.00m    1.86t
  [unknown]  pve            lvm2 a-m  <186.31g <186.31g       0
root@proxmox:~# vgreduce pve --removemissing
  WARNING: Couldn't find device with uuid miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T.
  WARNING: VG pve is missing PV miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T (last written to /dev/sdc).
  WARNING: Couldn't find device with uuid miedLh-X9u2-IbLk-xA35-HrGr-VEy7-8txs9T.
  Wrote out consistent volume group pve.
root@proxmox:~# pvs -o+pv_used
  PV         VG             Fmt  Attr PSize    PFree    Used 
  /dev/sda3  pve            lvm2 a--  <837.36g   16.00g <821.36g
  /dev/sdd   Daddy_Game_2tb lvm2 a--  <186.31g <186.31g       0
  /dev/sde   Daddy_Game_2tb lvm2 a--     1.86t  376.00m    1.86t

so looking at the output of # pvs -o+pv_used (above showing 376.00m free) I think maybe adding the 100gb hdd to Daddy_Game_2TB could help me get it booted long enough to make a normal windows backup and maybe transfer my data out. We'll find out on tomorrow's episode of "some dumb guy bought a server on craigslist".

I hope someday someone reads this :)
 
Last edited:
https://christitus.com/lvm-guide/

Found this today at lunch. It really helped a beginner understand the layers at least.

https://www.reddit.com/r/Proxmox/comments/ow9zc7/lvm_vs_lvm_thin/

also incredibly helpful. apparently the lvm thin snapshot function is more like a bookmark than a backup. from what was explained in this article if you deleted a snapshot directly with lvremove you could loose all progress from the previous snapshot to that one?? not really sure yet, but sounds too dangerous to try :) ...until i get a proper backup of my data.
 
Last edited:
  • Like
Reactions: Kingneutron
New day, new info. This information was very helpful. Looks like I extended my 1.8tb ssd to 2750G. Bad me.

Code: 
root@proxmox:/etc/pve/qemu-server# cat 100.conf
args: -cpu host,kvm=off
bios: ovmf
boot: order=scsi0;ide0;ide2;net0
cores: 20
cpu: host
efidisk0: Daddy_Game_2tb:vm-100-disk-1,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:04:00,pcie=1
ide0: local:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
ide2: local:iso/Windows2.iso,media=cdrom,size=4779200K
machine: pc-q35-8.1,viommu=virtio
memory: 65536
meta: creation-qemu=8.1.5,ctime=1715402358
name: DaddyGame
net0: virtio=BC:24:11:C9:68:62,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
parent: memb4memu
scsi0: Daddy_Game_2tb:vm-100-disk-0,iothread=1,size=2750G
scsihw: virtio-scsi-single
smbios1: uuid=aea8e8f9-705f-4e8f-82f3-b674d1fc3af2
sockets: 2
usb0: host=04ca:004b
usb1: host=04b3:310c
vga: none
vmgenid: 63ffe7a4-63a5-48c3-9b70-fa4d71dd4ecd
vmstatestorage: Daddy_Game_2tb

I tried to resize using qemu commands with qm resize 100 Daddy_Game_2tb 2.7tand got :

Code: 
400 Parameter verification failed.
size: value does not match the regex pattern
disk: value 'Daddy_Game_2tb' does not have a value in the enumeration 'ide0, ide1, ide2, ide3, scsi0, scsi1, scsi2, scsi3, scsi4, scsi5, scsi6, scsi7, scsi8, scsi9, scsi10, scsi11, scsi12, scsi13, scsi14, scsi15, scsi16, scsi17, scsi18, scsi19, scsi20, scsi21, scsi22, scsi23, scsi24, scsi25, scsi26, scsi27, scsi28, scsi29, scsi30, virtio0, virtio1, virtio2, virtio3, virtio4, virtio5, virtio6, virtio7, virtio8, virtio9, virtio10, virtio11, virtio12, virtio13, virtio14, virtio15, sata0, sata1, sata2, sata3, sata4, sata5, efidisk0, tpmstate0'
qm disk resize <vmid> <disk> <size> [OPTIONS]

Possibly I need to use G as displayed above for qm command to work. I tried this as it looked like the safest way to get proxmox to overprovision back to big so I could get the data.
 
Tried to start the vm and it is showing .../...tmeta is active

Code: 
root@proxmox:/etc/lvm/archive# qm start 100
activating LV 'Daddy_Game_2tb/Daddy_Game_2tb' failed:   Activation of logical volume Daddy_Game_2tb/Daddy_Game_2tb is prohibited while logical volume Daddy_Game_2tb/Daddy_Game_2tb_tmeta is active.

But when I look at the lvscan it is not active:
Code: 
root@proxmox:/etc/lvm/archive# lvscan
  inactive          '/dev/Daddy_Game_2tb/Daddy_Game_2tb' [1.83 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/vm-100-disk-1' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/vm-100-state-New2TB' [<128.49 GiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-0_New2TB' [1.12 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-1_New2TB' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-0_withAsuka' [<2.69 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-1_withAsuka' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-0_memb4memu' [<2.69 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-1_memb4memu' [4.00 MiB] inherit
  ACTIVE            '/dev/pve/data' [703.01 GiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit

correction lvscan -a
Code: 
root@proxmox:/etc/lvm/archive# lvscan -a
  inactive          '/dev/Daddy_Game_2tb/Daddy_Game_2tb' [1.83 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/vm-100-disk-1' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/vm-100-state-New2TB' [<128.49 GiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-0_New2TB' [1.12 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-1_New2TB' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-0_withAsuka' [<2.69 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-1_withAsuka' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-0_memb4memu' [<2.69 TiB] inherit
  inactive          '/dev/Daddy_Game_2tb/snap_vm-100-disk-1_memb4memu' [4.00 MiB] inherit
  inactive          '/dev/Daddy_Game_2tb/lvol0_pmspare' [<15.88 GiB] inherit
  ACTIVE            '/dev/Daddy_Game_2tb/Daddy_Game_2tb_tmeta' [<15.88 GiB] inherit
  ACTIVE            '/dev/Daddy_Game_2tb/Daddy_Game_2tb_tdata' [1.83 TiB] inherit
  ACTIVE            '/dev/pve/data' [703.01 GiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  ACTIVE            '/dev/pve/data_tdata' [703.01 GiB] inherit
  ACTIVE            '/dev/pve/data_tmeta' [7.17 GiB] inherit
  inactive          '/dev/pve/lvol0_pmspare' [7.17 GiB] inherit

Extremely helpful for LVM activation/deactivation:
https://docs.redhat.com/en/document...tion-configuring-and-managing-logical-volumes

vgchange -an Daddy_Game_2tb
this deactivated the tmeta/tdata, but also resulted in vg Daddy_Game_2tb being unable to be seen. just hung on lvs/pvs until i hit ctl+c and then reported that lock could not be obtained.

found this:
https://serverfault.com/questions/383726/system-cant-get-lock-on-vg-after-exending-lv
 
Last edited:
  • Like
Reactions: Kingneutron
This could be very useful for someone if metada was not at a system limit (15.88 is the max allowed by lvm for any metada volume for some reason)

https://charlmert.github.io/blog/2017/06/15/lvm-metadata-repair/

I think I've reached the point where I will attempt to replace the metadata with the latest smaller one and pray. I know prayer isn't usually the best option with computer issues...but it cant hurt.
 
Last edited:
started going through possible backups...
vgcfgrestore --list Daddy_Game_2tb

root@proxmox:~# vgcfgrestore --list Daddy_Game_2tb

There were a bunch, the one that is very interesting is
Code: 
File:         /etc/lvm/archive/Daddy_Game_2tb_00018-711206966.vg/Daddy_Game_2tb_00018-711206966.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-0_withAsuka -pr -s Daddy_Game_2tb/vm-100-disk-0'
  Backup Time:  Fri Jun 21 23:02:04 2024

 
  File:         /etc/lvm/archive/Daddy_Game_2tb_00019-146088274.vg/Daddy_Game_2tb_00019-146088274.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-0_withAsuka -pr -s Daddy_Game_2tb/vm-100-disk-0'
  Backup Time:  Fri Jun 21 23:02:04 2024

 
  File:         /etc/lvm/archive/Daddy_Game_2tb_00020-1616929499.vg/Daddy_Game_2tb_00020-1616929499.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-1_withAsuka -pr -s Daddy_Game_2tb/vm-100-disk-1'
  Backup Time:  Fri Jun 21 23:02:04 2024

 
  File:         /etc/lvm/archive/Daddy_Game_2tb_00021-22902762.vg/Daddy_Game_2tb_00021-22902762.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-1_withAsuka -pr -s Daddy_Game_2tb/vm-100-disk-1'
  Backup Time:  Fri Jun 21 23:02:04 2024

This was the first snapshot I took after overprovisioning...has all data (although it may be corrupted or have moved when I resized the disk inside the vm to try and lvreduce back to reasonable size...but if i'm to restore this is the place to start)

Red Hat is a fantastic source of information on LVM. Many of the things I was looking for were clearly explained with examples explaining the CLI. This tells me the vgcfgrestore is not a stand alone command. It is ran after you restore the metadata with
Code: 
# pvcreate --uuid physical-volume-uuid \
           --restorefile /etc/lvm/archive/volume-group-name_backup-number.vg \
           block-device
https://docs.redhat.com/fr/document...on-an-lvm-physical-volume_troubleshooting-lvm

And it looks like my UUID is still in place because disk is still in the system...so this should do the trick. for the first part.
Code: 
pvcreate --restorefile /etc/lvm/archive/volume-group-name_backup-number.vg \
           block-device
 
Last edited:
so after a week of research and learning a lot about lvm-thin I was ready to overwrite the metadata. I was scared and decided to look at software options first. Found ReclaiMe, did a little research about it & it's used by data recovery professionals. Anyway, yanked the drive (luckily for me I hadn't spanned multiple drives too in this mess up). ReclaiMe is able to access the files/folders on the underlying NTFS partition. I am going to back it up, and then attempt the lvm recovery for fun...with much less pressure. Just wanted to put this here as a lifeline in case anybody else is screwed and just wants their data. will also update the main post so it is easy to find.
 
You may want to switch to ZFS after getting back to a stable state. LVM is a bit more of a "black box" and ZFS has more features - such as being more simple to admin, inline fast compression, native snapshots, easy builtin mirroring capability, send/recv, and per-dataset properties

https://github.com/kneutron/ansitest/tree/master/ZFS

https://github.com/kneutron/ansitest/tree/master/proxmox

And yes, don't forget to setup regular backups - preferably to proxmox backup server so you can take advantage of dedup and other features
 
Kingneutron said:
You may want to switch to ZFS after getting back to a stable state. LVM is a bit more of a "black box" and ZFS has more features - such as being more simple to admin, inline fast compression, native snapshots, easy builtin mirroring capability, send/recv, and per-dataset properties

https://github.com/kneutron/ansitest/tree/master/ZFS

https://github.com/kneutron/ansitest/tree/master/proxmox

And yes, don't forget to setup regular backups - preferably to proxmox backup server so you can take advantage of dedup and other features
Click to expand...
Thank you for your reply. I will look into it. LVM may be a very powerful tool for someone managing storage that changes rapidly for multiple users...but in my case a fixed format with more steps to extend would have been a better option. :)

I will look at ZFS Thank you again
 
None of the commands worked because every time I would run lvchange -an ...tmeta or tdata they would immediately start running thin_check and anything tried with that vg would fail because lock could not be obtained.

So I'm going nuclear and running:
lvchange -an (for both tdata and tmeta mounts)
...successful
lvremove Daddy_Game_2tb
...successful
vgcfgrestore Daddy_Game_2tb --force
"are you sure" Yes i'm friggin sure
...successful

now trying to recreate the logical volumes
rebooted
...nothing
Went back further
Code: 
root@proxmox:~# vgcfgrestore -l Daddy_Game_2tb
  File:         /etc/lvm/archive/Daddy_Game_2tb_00027-20100305.vg/Daddy_Game_2tb_00027-20100305.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-0_memb4memu -pr -s Daddy_Game_2tb/vm-100-disk-0'
  Backup Time:  Thu Jun 27 16:28:54 2024


 
  File:         /etc/lvm/archive/Daddy_Game_2tb_00028-464049504.vg/Daddy_Game_2tb_00028-464049504.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-1_memb4memu -pr -s Daddy_Game_2tb/vm-100-disk-1'
  Backup Time:  Thu Jun 27 16:28:54 2024


 
  File:         /etc/lvm/archive/Daddy_Game_2tb_00029-566202664.vg/Daddy_Game_2tb_00029-566202664.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-1_memb4memu -pr -s Daddy_Game_2tb/vm-100-disk-1'
  Backup Time:  Thu Jun 27 16:28:54 2024

root@proxmox:~# vgcfgrestore -f /etc/lvm/archive/Daddy_Game_2tb_00028-464049504.vg Daddy_Game_2tb --force
  WARNING: Forced restore of Volume Group Daddy_Game_2tb with thin volumes.
  Restored volume group Daddy_Game_2tb.

I didn't know which file to chose, so I just picked one. Probably kind of a big deal...but at this point it's lost to me and I am just experimenting.
cat of the metadata
Code: 
description = "Created *before* executing '/sbin/lvcreate -n snap_vm-100-disk-1_memb4memu -pr -s Daddy_Game_2tb/vm-100-disk-1'"


creation_host = "proxmox"       # Linux proxmox 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64
creation_time = 1719530934      # Thu Jun 27 16:28:54 2024


Daddy_Game_2tb {
        id = "5J6TDX-cLhZ-Ad5W-ftoF-cOyb-v0XB-492CrV"
        seqno = 29
        format = "lvm2"                 # informational
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0
        metadata_copies = 0


        physical_volumes {


                pv0 {
                        id = "AaGe4g-Rk9x-Mnwc-Yd8v-EpsX-IGGH-1dAYgr"
                        device = "/dev/sdc"     # Hint only


                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 4000797360   # 1.86302 Terabytes
                        pe_start = 2048
                        pe_count = 488378       # 1.86301 Terabytes
                }
        }


        logical_volumes {


                Daddy_Game_2tb {
                        id = "jEsFkK-cTgQ-c2zM-Nw63-KvJL-FMuT-yF01vO"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718902422      # 2024-06-20 09:53:42 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 480154   # 1.83164 Terabytes


                                type = "thin-pool+CROP_METADATA"
                                metadata = "Daddy_Game_2tb_tmeta"
                                pool = "Daddy_Game_2tb_tdata"
                                transaction_id = 12
                                chunk_size = 128        # 64 Kilobytes
                                discards = "passdown"
                                zero_new_blocks = 1
                                crop_metadata = 0
                        }
                }


                vm-100-disk-0 {
                        id = "jwQ12G-ezIV-SKo7-ssiQ-w14x-rYpN-GaJ7tm"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718903618      # 2024-06-20 10:13:38 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 704000   # 2.68555 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 0
                                device_id = 1
                        }
                }


                vm-100-disk-1 {
                        id = "bSWrYe-eaKy-Yyoe-b8qi-qGFr-RQyV-UiYJRi"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718917857      # 2024-06-20 14:10:57 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1        # 4 Megabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 3
                                device_id = 2
                        }
                }


                vm-100-state-New2TB {
                        id = "elmzLX-XwFe-ocV4-V79x-66yz-Y4h8-XhxAeW"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718918264      # 2024-06-20 14:17:44 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 32893    # 128.488 Gigabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 4
                                device_id = 3
                        }
                }


                snap_vm-100-disk-0_New2TB {
                        id = "bt5OJ1-Xc77-rjaC-9vyE-tNv8-kSfd-eEnac6"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1718918302      # 2024-06-20 14:18:22 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 294400   # 1.12305 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 5
                                device_id = 4
                                origin = "vm-100-disk-0"
                        }
                }


                snap_vm-100-disk-1_New2TB {
                        id = "KDyUyh-eb8d-bkTs-MKQZ-dvlA-OJM5-5ntc5P"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1718918302      # 2024-06-20 14:18:22 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1        # 4 Megabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 6
                                device_id = 5
                                origin = "vm-100-disk-1"
                        }
                }


                snap_vm-100-disk-0_withAsuka {
                        id = "ngzIIF-jLe6-XnTg-JX1o-s4Ea-Lue7-0iqU8H"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1719036124      # 2024-06-21 23:02:04 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 704000   # 2.68555 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 7
                                device_id = 6
                                origin = "vm-100-disk-0"
                        }
                }


                snap_vm-100-disk-1_withAsuka {
                        id = "34gk9x-70iK-MgQH-rHvC-AUqv-3QoD-N4fHWl"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1719036124      # 2024-06-21 23:02:04 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1        # 4 Megabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 8
                                device_id = 7
                                origin = "vm-100-disk-1"
                        }
                }


                snap_vm-100-disk-0_memb4memu {
                        id = "07LLJk-HyOy-z7f0-UW0i-r34T-texW-hKX2xS"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1719530934      # 2024-06-27 16:28:54 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 704000   # 2.68555 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 11
                                device_id = 8
                                origin = "vm-100-disk-0"
                        }
                }


                lvol0_pmspare {
                        id = "dX1Va1-bosp-0L1z-qbNK-PE04-r0cL-rme00s"
                        status = ["READ", "WRITE"]
                        flags = []
                        creation_time = 1718901995      # 2024-06-20 09:46:35 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 4065     # 15.8789 Gigabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }


                Daddy_Game_2tb_tmeta {
                        id = "8oeWxv-shQW-lHCX-aIyr-Jzt9-ARNt-WdL5uQ"
                        status = ["READ", "WRITE"]
                        flags = []
                        creation_time = 1718902849      # 2024-06-20 10:00:49 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 4065     # 15.8789 Gigabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 484219
                                ]
                        }
                }


                Daddy_Game_2tb_tdata {
                        id = "cwDeQa-5gx5-qLPw-Br8c-MfCV-gYrB-3JceeF"
                        status = ["READ", "WRITE"]
                        flags = []
                        creation_time = 1718902849      # 2024-06-20 10:00:49 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 480154   # 1.83164 Terabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 4065
                                ]
                        }
                }
        }


}

This has the UUIDs of everything in it...so i'm guessing it was the/a right one.

Then...
Code: 
root@proxmox:/etc/lvm/archive# pvs -a
  WARNING: ignoring metadata seqno 30 on /dev/sde for seqno 37 on /dev/sdd for VG Daddy_Game_2tb.
  WARNING: Inconsistent metadata found for VG Daddy_Game_2tb.
  See vgck --updatemetadata to correct inconsistency.
  PV         VG             Fmt  Attr PSize    PFree  
  /dev/sda2                      ---        0        0
  /dev/sda3  pve            lvm2 a--  <837.36g   16.00g
  /dev/sdb1                      ---        0        0
  /dev/sdd   Daddy_Game_2tb lvm2 a--  <186.31g <186.31g
  /dev/sde   Daddy_Game_2tb lvm2 a--     1.86t    1.86t

Lets see if lvm2 led us astray
Code: 
root@proxmox:/etc/lvm/archive# vgck --updatemetadata Daddy_Game_2tb
  WARNING: ignoring metadata seqno 30 on /dev/sde for seqno 37 on /dev/sdd for VG Daddy_Game_2tb.
  WARNING: Inconsistent metadata found for VG Daddy_Game_2tb.
  See vgck --updatemetadata to correct inconsistency.
  WARNING: updating old metadata to 38 on /dev/sde for VG Daddy_Game_2tb.
root@proxmox:/etc/lvm/archive# pvs
  PV         VG             Fmt  Attr PSize    PFree  
  /dev/sda3  pve            lvm2 a--  <837.36g   16.00g
  /dev/sdd   Daddy_Game_2tb lvm2 a--  <186.31g <186.31g
  /dev/sde   Daddy_Game_2tb lvm2 a--     1.86t    1.86t
so it shows up now as a vg again without error...but will my lv be there? ...
Code: 
root@proxmox:/etc/lvm/archive# lvs -a
  LV              VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data            pve twi-a-tz-- 703.01g             0.00   0.26                          
  [data_tdata]    pve Twi-ao---- 703.01g                                                  
  [data_tmeta]    pve ewi-ao----   7.17g                                                  
  [lvol0_pmspare] pve ewi-------   7.17g                                                  
  root            pve -wi-ao----  96.00g                                                  
  swap            pve -wi-ao----   8.00g

not so hot :) ... try again maybe
 
Last edited:
...condinued
Getting the current UUIDs
Code: 
root@proxmox:~# pvs -av
  PV         VG             Fmt  Attr PSize    PFree    DevSize  PV UUID                             
  /dev/sda2                      ---        0        0     1.00g                                     
  /dev/sda3  pve            lvm2 a--  <837.36g   16.00g  837.36g Jf42Tj-jZpb-cGSU-3lPM-8hwx-IAq4-TQ91IH
  /dev/sdb1                      ---        0        0  <953.87g                                     
  /dev/sdd   Daddy_Game_2tb lvm2 a--     1.86t    1.86t    1.86t AaGe4g-Rk9x-Mnwc-Yd8v-EpsX-IGGH-1dAYgr
  /dev/sde   Daddy_Game_2tb lvm2 a--  <186.31g <186.31g  186.31g eopRb8-vfSD-Frmh-4edN-NThs-6X93-YbfmGo

UUID /dev/sdd seems to match above archive file
Code: 
physical_volumes {


                pv0 {
                        id = "AaGe4g-Rk9x-Mnwc-Yd8v-EpsX-IGGH-1dAYgr"
                        device = "/dev/sdc"     # Hint only


                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 4000797360   # 1.86302 Terabytes
                        pe_start = 2048
                        pe_count = 488378       # 1.86301 Terabytes
                }
        }

but no LV shows up
Code: 
root@proxmox:~# lvscan -a
  ACTIVE            '/dev/pve/data' [703.01 GiB] inherit
  ACTIVE            '/dev/pve/swap' [8.00 GiB] inherit
  ACTIVE            '/dev/pve/root' [96.00 GiB] inherit
  ACTIVE            '/dev/pve/data_tdata' [703.01 GiB] inherit
  ACTIVE            '/dev/pve/data_tmeta' [7.17 GiB] inherit
  inactive          '/dev/pve/lvol0_pmspare' [7.17 GiB] inherit

This is after a reboot as well. Trying to get them to show up even if they aren't active is more of a headache than I suspected. Maybe I need to go back further since I deleted (lvremove) the memb4memu snapshot and it exists in the archive file I am using.

vgcfgrestore -l Daddy_Game_2tb gives me the next newest one

Code: 
File:         /etc/lvm/archive/Daddy_Game_2tb_00022-792151129.vg/Daddy_Game_2tb_00022-792151129.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -aly -V 134729728k --name vm-100-state-memupud --thinpool Daddy_Game_2tb/Daddy_Game_2tb'
  Backup Time:  Thu Jun 27 16:20:40 2024


 
  File:         /etc/lvm/archive/Daddy_Game_2tb_00023-1389181989.vg/Daddy_Game_2tb_00023-1389181989.vg
  VG name:      Daddy_Game_2tb
  Description:  Created *before* executing '/sbin/lvcreate -aly -V 134729728k --name vm-100-state-memupud --thinpool Daddy_Game_2tb/Daddy_Game_2tb'
  Backup Time:  Thu Jun 27 16:20:40 2024

Lets peek inside
Code: 
root@proxmox:/etc/lvm/archive# cat /etc/lvm/archive/Daddy_Game_2tb_00022-792151129.vg
# Generated by LVM2 version 2.03.16(2) (2022-05-18): Thu Jun 27 16:20:40 2024


contents = "Text Format Volume Group"
version = 1


description = "Created *before* executing '/sbin/lvcreate -aly -V 134729728k --name vm-100-state-memupud --thinpool Daddy_Game_2tb/Daddy_Game_2tb'"


creation_host = "proxmox"       # Linux proxmox 6.8.4-2-pve #1 SMP PREEMPT_DYNAMIC PMX 6.8.4-2 (2024-04-10T17:36Z) x86_64
creation_time = 1719530440      # Thu Jun 27 16:20:40 2024


Daddy_Game_2tb {
        id = "5J6TDX-cLhZ-Ad5W-ftoF-cOyb-v0XB-492CrV"
        seqno = 23
        format = "lvm2"                 # informational
        status = ["RESIZEABLE", "READ", "WRITE"]
        flags = []
        extent_size = 8192              # 4 Megabytes
        max_lv = 0
        max_pv = 0
        metadata_copies = 0


        physical_volumes {


                pv0 {
                        id = "AaGe4g-Rk9x-Mnwc-Yd8v-EpsX-IGGH-1dAYgr"
                        device = "/dev/sdc"     # Hint only


                        status = ["ALLOCATABLE"]
                        flags = []
                        dev_size = 4000797360   # 1.86302 Terabytes
                        pe_start = 2048
                        pe_count = 488378       # 1.86301 Terabytes
                }
        }


        logical_volumes {


                Daddy_Game_2tb {
                        id = "jEsFkK-cTgQ-c2zM-Nw63-KvJL-FMuT-yF01vO"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718902422      # 2024-06-20 09:53:42 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 480154   # 1.83164 Terabytes


                                type = "thin-pool+CROP_METADATA"
                                metadata = "Daddy_Game_2tb_tmeta"
                                pool = "Daddy_Game_2tb_tdata"
                                transaction_id = 9
                                chunk_size = 128        # 64 Kilobytes
                                discards = "passdown"
                                zero_new_blocks = 1
                                crop_metadata = 0
                        }
                }


                vm-100-disk-0 {
                        id = "jwQ12G-ezIV-SKo7-ssiQ-w14x-rYpN-GaJ7tm"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718903618      # 2024-06-20 10:13:38 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 704000   # 2.68555 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 0
                                device_id = 1
                        }
                }


                vm-100-disk-1 {
                        id = "bSWrYe-eaKy-Yyoe-b8qi-qGFr-RQyV-UiYJRi"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718917857      # 2024-06-20 14:10:57 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1        # 4 Megabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 3
                                device_id = 2
                        }
                }


                vm-100-state-New2TB {
                        id = "elmzLX-XwFe-ocV4-V79x-66yz-Y4h8-XhxAeW"
                        status = ["READ", "WRITE", "VISIBLE"]
                        flags = []
                        creation_time = 1718918264      # 2024-06-20 14:17:44 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 32893    # 128.488 Gigabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 4
                                device_id = 3
                        }
                }


                snap_vm-100-disk-0_New2TB {
                        id = "bt5OJ1-Xc77-rjaC-9vyE-tNv8-kSfd-eEnac6"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1718918302      # 2024-06-20 14:18:22 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 294400   # 1.12305 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 5
                                device_id = 4
                                origin = "vm-100-disk-0"
                        }
                }


                snap_vm-100-disk-1_New2TB {
                        id = "KDyUyh-eb8d-bkTs-MKQZ-dvlA-OJM5-5ntc5P"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1718918302      # 2024-06-20 14:18:22 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1        # 4 Megabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 6
                                device_id = 5
                                origin = "vm-100-disk-1"
                        }
                }


                snap_vm-100-disk-0_withAsuka {
                        id = "ngzIIF-jLe6-XnTg-JX1o-s4Ea-Lue7-0iqU8H"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1719036124      # 2024-06-21 23:02:04 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 704000   # 2.68555 Terabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 7
                                device_id = 6
                                origin = "vm-100-disk-0"
                        }
                }


                snap_vm-100-disk-1_withAsuka {
                        id = "34gk9x-70iK-MgQH-rHvC-AUqv-3QoD-N4fHWl"
                        status = ["READ", "VISIBLE"]
                        flags = ["ACTIVATION_SKIP"]
                        creation_time = 1719036124      # 2024-06-21 23:02:04 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 1        # 4 Megabytes


                                type = "thin"
                                thin_pool = "Daddy_Game_2tb"
                                transaction_id = 8
                                device_id = 7
                                origin = "vm-100-disk-1"
                        }
                }


                lvol0_pmspare {
                        id = "dX1Va1-bosp-0L1z-qbNK-PE04-r0cL-rme00s"
                        status = ["READ", "WRITE"]
                        flags = []
                        creation_time = 1718901995      # 2024-06-20 09:46:35 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 4065     # 15.8789 Gigabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 0
                                ]
                        }
                }


                Daddy_Game_2tb_tmeta {
                        id = "8oeWxv-shQW-lHCX-aIyr-Jzt9-ARNt-WdL5uQ"
                        status = ["READ", "WRITE"]
                        flags = []
                        creation_time = 1718902849      # 2024-06-20 10:00:49 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 4065     # 15.8789 Gigabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 484219
                                ]
                        }
                }


                Daddy_Game_2tb_tdata {
                        id = "cwDeQa-5gx5-qLPw-Br8c-MfCV-gYrB-3JceeF"
                        status = ["READ", "WRITE"]
                        flags = []
                        creation_time = 1718902849      # 2024-06-20 10:00:49 -0700
                        creation_host = "proxmox"
                        segment_count = 1


                        segment1 {
                                start_extent = 0
                                extent_count = 480154   # 1.83164 Terabytes


                                type = "striped"
                                stripe_count = 1        # linear


                                stripes = [
                                        "pv0", 4065
                                ]
                        }
                }
        }


}

It appears to have everything and the pv UUID is a match...lets spin the wheel.
Code: 
root@proxmox:/etc/lvm/archive# vgcfgrestore Daddy_Game_2tb -f /etc/lvm/archive/Daddy_Game_2tb_00022-792151129.vg --force
  WARNING: Forced restore of Volume Group Daddy_Game_2tb with thin volumes.
  Restored volume group Daddy_Game_2tb.
root@proxmox:/etc/lvm/archive# pvs -av
  WARNING: ignoring metadata seqno 24 on /dev/sdd for seqno 39 on /dev/sde for VG Daddy_Game_2tb.
  WARNING: Inconsistent metadata found for VG Daddy_Game_2tb.
  See vgck --updatemetadata to correct inconsistency.
  PV         VG             Fmt  Attr PSize    PFree    DevSize  PV UUID                             
  /dev/sda2                      ---        0        0     1.00g                                     
  /dev/sda3  pve            lvm2 a--  <837.36g   16.00g  837.36g Jf42Tj-jZpb-cGSU-3lPM-8hwx-IAq4-TQ91IH
  /dev/sdb1                      ---        0        0  <953.87g                                     
  /dev/sdd   Daddy_Game_2tb lvm2 a--     1.86t    1.86t    1.86t AaGe4g-Rk9x-Mnwc-Yd8v-EpsX-IGGH-1dAYgr
  /dev/sde   Daddy_Game_2tb lvm2 a--  <186.31g <186.31g  186.31g eopRb8-vfSD-Frmh-4edN-NThs-6X93-YbfmGo

It's the same and I believe last time it updated metadata to 39? not sure what that means but it sounds like it changing the metadata back every time I vgcfgrestore it. Beginning to wonder if the metadata locations have changed and I possibly could just be corrupting data each time I write the archive? Well...that's what figuring stuff out looks like sometimes. Getting close to giving up and wiping.

...maybe I should have started with thin_dump :)
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom