LVM Thin Pool Issue

[thebuggy83]

Dear Forum Members,
I have Proxmox running on LVM Thin Pools. As I know now I made many mistakes. I did not monitor or extend poolmetadatasize.
I did not backup the lvm archives regularily.
Therefore I think I have a broken Metadata issue.

At the moment I have the situation that my Volume Group "vga" has root and swap on it which is working.
On the same Volume Group "vda" I have a Thin Pool named "data-hdd".
In this thin pool there are many logical thin volumes.

Here the output of lvscan

root@citprx01 /usr/src/thin-provisioning-tools/doc/thinp-version-2 # lvscan
ACTIVE '/dev/vdb/data-ssd' [400.00 GiB] inherit
ACTIVE '/dev/vda/root' [50.00 GiB] inherit
ACTIVE '/dev/vda/swap' [4.00 GiB] inherit
inactive '/dev/vda/data-hdd' [1.95 TiB] inherit
inactive '/dev/vda/vm-118-disk-0' [8.00 GiB] inherit
inactive '/dev/vda/vm-119-disk-0' [32.00 GiB] inherit
inactive '/dev/vda/vm-124-disk-0' [30.00 GiB] inherit
inactive '/dev/vda/vm-125-disk-0' [26.00 GiB] inherit
inactive '/dev/vda/vm-114-disk-0' [40.00 GiB] inherit
inactive '/dev/vda/vm-115-disk-0' [8.00 GiB] inherit
inactive '/dev/vda/vm-121-disk-0' [20.00 GiB] inherit
inactive '/dev/vda/vm-122-disk-0' [65.00 GiB] inherit
inactive '/dev/vda/snap_vm-115-disk-0_vzdump' [8.00 GiB] inherit
ACTIVE '/dev/vda/data-hdd_meta0' [128.00 MiB] inherit
ACTIVE '/dev/vda/data-hdd_meta1' [128.00 MiB] inherit
ACTIVE '/dev/vda/temp_meta' [128.00 MiB] inherit
ACTIVE '/dev/vda/meta_data-hdd' [500.00 MiB] inherit
root@citprx01 /usr/src/thin-provisioning-tools/doc/thinp-version-2 #

and lvs -a

root@citprx01 /usr/src/thin-provisioning-tools/doc/thinp-version-2 # lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data-hdd vda twi---tz-- 1.95t
data-hdd_meta0 vda -wi------- 128.00m
data-hdd_meta1 vda -wi------- 128.00m
[data-hdd_tdata] vda Twi------- 1.95t
[data-hdd_tmeta] vda ewi------- 512.00m
[lvol2_pmspare] vda ewi------- 128.00m
meta_data-hdd vda -wi-a----- 500.00m
root vda -wi-ao---- 50.00g
snap_vm-115-disk-0_vzdump vda Vri---tz-k 8.00g data-hdd vm-115-disk-0
swap vda -wi-ao---- 4.00g
temp_meta vda -wi------- 128.00m
vm-114-disk-0 vda Vwi---tz-- 40.00g data-hdd
vm-115-disk-0 vda Vwi---tz-- 8.00g data-hdd
vm-118-disk-0 vda Vwi---tz-- 8.00g data-hdd
vm-119-disk-0 vda Vwi---tz-- 32.00g data-hdd
vm-121-disk-0 vda Vwi---tz-- 20.00g data-hdd
vm-122-disk-0 vda Vwi---tz-- 65.00g data-hdd
vm-124-disk-0 vda Vwi---tz-- 30.00g data-hdd
vm-125-disk-0 vda Vwi---tz-- 26.00g data-hdd
data-ssd vdb twi-a-tz-- 400.00g 0.00 10.43
[data-ssd_tdata] vdb Twi-ao---- 400.00g
[data-ssd_tmeta] vdb ewi-ao---- 100.00m
[lvol0_pmspare] vdb ewi------- 100.00m
root@citprx01 /usr/src/thin-provisioning-tools/doc/thinp-version-2 #

When I try to activate the Volumes:

root@citprx01 /usr/src/thin-provisioning-tools/doc/thinp-version-2 # lvchange -ay vda
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.
Thin pool vda-data--hdd-tpool (253:8) transaction_id is 4542, while expected 4544.

I already compiled rust and thin-provisioning-tools on my machine and hoped to restore the metadata with thin_scan. Unfortunately I was not able to change the features that the binary was created, although I used the feature "devtools"

Maybe you can answer one or more of my questions.

1. Is the way with thin_scan the correct to resolve my issue?
2. and if which feature flag I have to set that it is built?
3. do you have any other suggesstions for me at this point?

Thanks in advance.
Markus

 

[thebuggy83]

With the help of mingnus, developer of the thin-provisioning-tools we solved the problem the following way:

root@citprx01 ~ # lvchange -ay vda/data-hdd_tmeta -y -f
Allowing activation of component LV.

root@citprx01 ~/.cargo/bin # pdata_tools thin_check /dev/mapper/vda-data--hdd_tmeta
examining superblock
examining devices tree
examining mapping tree
checking space map counts

I also exported the metadata into xml with this command:
root@citprx01 ~ # ./pdata_tools thin_dump -m /dev/mapper/vda-data--hdd_tmeta -o /root/metadata-data_hdd.xml
and attached it.
metadata-data_hdd-xml.txt

root@citprx01 ~ # vgcfgrestore --force vda -f vda.lvm
Volume group vda has active volume: data-hdd_meta0.
Volume group vda has active volume: data-hdd_tmeta.
Volume group vda has active volume: temp_meta.
Volume group vda has active volume: meta_data-hdd.
Volume group vda has active volume: swap.
Volume group vda has active volume: root.
Volume group vda has active volume: data-hdd_meta1.
WARNING: Found 7 active volume(s) in volume group "vda".
Restoring VG with active LVs, may cause mismatch with its meta
data.
Do you really want to proceed with restore of volume group "vda", while 7 volume(s) are active? [y/n]: y
WARNING: Forced restore of Volume Group vda with thin volumes.
Restored volume group vda.
root@citprx01 ~ #

root@citprx01 ~ # lvchange -an vda/data-hdd_tmeta
root@citprx01 ~ # vgchange -ay
1 logical volume(s) in volume group "vdb" now active
15 logical volume(s) in volume group "vda" now active
root@citprx01 ~ #

One further question:
When I check lvs -a pool metadata is not full:

root@citprx01 ~ # lvs -a
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
data-hdd vda twi-aotz-- 1.95t 10.07 4.49
data-hdd_meta0 vda -wi-a----- 128.00m
data-hdd_meta1 vda -wi-a----- 128.00m
[data-hdd_tdata] vda Twi-ao---- 1.95t
[data-hdd_tmeta] vda ewi-ao---- 512.00m
[lvol2_pmspare] vda ewi------- 128.00m
meta_data-hdd vda -wi-a----- 500.00m
root vda -wi-ao---- 50.00g
swap vda -wi-ao---- 4.00g
temp_meta vda -wi-a----- 128.00m
vm-114-disk-0 vda Vwi-a-tz-- 40.00g data-hdd 99.92
vm-115-disk-0 vda Vwi-a-tz-- 8.00g data-hdd 20.24
vm-118-disk-0 vda Vwi-a-tz-- 8.00g data-hdd 78.49
vm-119-disk-0 vda Vwi-a-tz-- 32.00g data-hdd 96.33
vm-121-disk-0 vda Vwi-a-tz-- 20.00g data-hdd 21.50
vm-122-disk-0 vda Vwi-a-tz-- 65.00g data-hdd 97.88
vm-124-disk-0 vda Vwi-a-tz-- 30.00g data-hdd 99.71
vm-125-disk-0 vda Vwi-a-tz-- 26.00g data-hdd 95.76
data-ssd vdb twi-a-tz-- 400.00g 0.00 10.43
[data-ssd_tdata] vdb Twi-ao---- 400.00g
[data-ssd_tmeta] vdb ewi-ao---- 100.00m
[lvol0_pmspare] vdb ewi------- 100.00m
root@citprx01 ~ #

Maybe it was this old snapshot which did this hell thing?
But I read about something that you can tell the pool it should autoresize itself?

How can I thank you for you help, I was really desperate at the time when I wrote this post yesterday. May I support any of your projects?

Best regards
Markus