[SOLVED] Restore overflown lvm - just one small step, I'm sure...

[soulsurfer42]

Dear Forum,

after several days, I admit defeat. I upgraded ProxMox VE from 5.4 to 6 to 7 and made a snapshot of the most vital VM beforehand. That was too much for the LVM, which ceased to work a couple of days later because it ran out of space (presumably).

I have been trying to restore the LVM ever since. What worked: I added a disk, created a new PV and extended the Volume Group so that it now holds enough free space:

root@ute:/dev/mapper# vgs -a VG #PV #LV #SN Attr VSize VFree pve 2 36 0 wz--n- 10.00t <704.00g

What did not work: Extending the relevant Logical Volume and it's corresponding meta data via

lvresize --size +700g --poolmetadatasize +2g pve/data

I fiddled around with lvchange --repair and the creation of a larger LV (18GB) to hold the meta data. The only result was, however, that activation of pve/data (the relevant LV) fails - but it's "intestines" do not: I can activate "pve/data_tdata" and "pve/data_tmeta", the latter in read-only mode.

Running "vgcfgrestore pve --test" gave a warning because the system is running from the logical volume "root" within the same VG. It failed, and so did my attempts to copy /etc/lvm to a live system and trying it from there with completely inactive LVs (I could not chroot for obvious reasons because the system partition is within the bloody LVM I'm trying to fix).

I feel I can't be far off. The matter is both urgent and important. Any help is greatly appreciatetd.

Best regards


Christian


PS:

This is what happens:

root@ute:/dev/mapper# lvchange -ay -v pve/data Activating logical volume pve/data. Activation of logical volume pve/data is prohibited while logical volume pve/data_tdata is active. root@ute:/dev/mapper# lvchange -an -v pve/data_tdata Deactivating logical volume pve/data_tdata. Removing pve-data_tdata (253:2) root@ute:/dev/mapper# lvchange -ay -v pve/data Activating logical volume pve/data. activation/volume_list configuration setting not defined: Checking only host tags for pve/data. Creating pve-data_tmeta Loading table for pve-data_tmeta (253:2). device-mapper: reload ioctl on (253:2) failed: Invalid argument Removing pve-data_tmeta (253:2) Creating pve-data_tdata Loading table for pve-data_tdata (253:2). Resuming pve-data_tdata (253:2).



lvscan -a gives the following (and yes, I did indeed try for quite a long time):

root@ute:/dev/mapper# lvscan -a ACTIVE '/dev/pve/swap' [8.00 GiB] inherit ACTIVE '/dev/pve/root' [96.00 GiB] inherit inactive '/dev/pve/data' [<7.13 TiB] inherit inactive '/dev/pve/vm-115-disk-1' [100.00 GiB] inherit inactive '/dev/pve/vm-112-disk-1' [100.00 GiB] inherit inactive '/dev/pve/vm-104-disk-1' [300.00 GiB] inherit inactive '/dev/pve/vm-104-disk-2' [32.00 GiB] inherit inactive '/dev/pve/vm-106-disk-1' [500.00 GiB] inherit inactive '/dev/pve/vm-102-disk-0' [250.00 GiB] inherit inactive '/dev/pve/vm-102-disk-1' [1.46 TiB] inherit inactive '/dev/pve/vm-400-state-preupdate' [<32.49 GiB] inherit inactive '/dev/pve/snap_vm-400-disk-0_preupdate' [1.46 TiB] inherit inactive '/dev/pve/snap_vm-400-disk-1_preupdate' [250.00 GiB] inherit inactive '/dev/pve/base-101-disk-0' [100.00 GiB] inherit inactive '/dev/pve/vm-100-disk-0' [200.00 GiB] inherit inactive '/dev/pve/vm-100-state-prenetworkreconfig' [<8.49 GiB] inherit inactive '/dev/pve/snap_vm-100-disk-0_prenetworkreconfig' [200.00 GiB] inherit inactive '/dev/pve/vm-102-disk-2' [100.00 GiB] inherit inactive '/dev/pve/snap_vm-400-disk-0_prerestore' [1.46 TiB] inherit inactive '/dev/pve/snap_vm-400-disk-1_prerestore' [250.00 GiB] inherit inactive '/dev/pve/vm-400-disk-0' [1.46 TiB] inherit inactive '/dev/pve/vm-400-disk-1' [250.00 GiB] inherit inactive '/dev/pve/snap_vm-102-disk-2_Neuinstallation' [100.00 GiB] inherit inactive '/dev/pve/vm-103-disk-0' [300.00 GiB] inherit inactive '/dev/pve/vm-102-state-Preinstallation' [<8.49 GiB] inherit inactive '/dev/pve/snap_vm-102-disk-2_Preinstallation' [100.00 GiB] inherit inactive '/dev/pve/vm-105-disk-0' [100.00 GiB] inherit inactive '/dev/pve/vm-108-disk-0' [1.46 TiB] inherit inactive '/dev/pve/vm-109-disk-0' [150.00 GiB] inherit inactive '/dev/pve/snap_vm-400-disk-0_preupdate_grooooooss' [1.46 TiB] inherit inactive '/dev/pve/snap_vm-400-disk-1_preupdate_grooooooss' [250.00 GiB] inherit ACTIVE '/dev/pve/fredegund' [2.00 TiB] inherit ACTIVE '/dev/pve/data_meta1' [16.00 GiB] inherit ACTIVE '/dev/pve/data_meta0' [16.00 GiB] inherit ACTIVE '/dev/pve/repaired_01' [4.00 GiB] inherit ACTIVE '/dev/pve/data_meta2' [18.00 GiB] inherit ACTIVE '/dev/pve/data_tdata' [<7.13 TiB] inherit inactive '/dev/pve/data_tmeta' [18.00 GiB] inherit inactive '/dev/pve/lvol3_pmspare' [18.00 GiB] inherit

lsblk:

root@ute:/dev/mapper# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 7.3T 0 disk ├─sda1 8:1 0 1007K 0 part ├─sda2 8:2 0 512M 0 part └─sda3 8:3 0 7.3T 0 part ├─pve-swap 253:0 0 8G 0 lvm [SWAP] ├─pve-root 253:1 0 96G 0 lvm / ├─pve-data_tdata 253:2 0 7.1T 0 lvm ├─pve-data_meta1 253:4 0 16G 0 lvm ├─pve-data_meta0 253:5 0 16G 0 lvm └─pve-data_meta2 253:7 0 18G 0 lvm sdb 8:16 0 2.7T 0 disk ├─pve-fredegund 253:3 0 2T 0 lvm ├─pve-repaired_01 253:6 0 4G 0 lvm └─pve-data_meta2 253:7 0 18G 0 lvm sdc 8:32 1 3.8G 0 disk ├─sdc1 8:33 1 2.9G 0 part ├─sdc2 8:34 1 3.9M 0 part └─sdc3 8:35 1 970M 0 part sr0 11:0 1 479.1M 0 rom

lvs -a:

root@ute:/dev/mapper# lvs -a LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert base-101-disk-0 pve Vri---tz-k 100.00g data data pve twi---tz-- <7.13t data_meta0 pve -wi-a----- 16.00g data_meta1 pve -wi-a----- 16.00g data_meta2 pve -wi-a----- 18.00g [data_tdata] pve Twi-a----- <7.13t [data_tmeta] pve ewi------- 18.00g fredegund pve -wi-a----- 2.00t [lvol3_pmspare] pve ewi------- 18.00g repaired_01 pve -wi-a----- 4.00g root pve -wi-ao---- 96.00g snap_vm-100-disk-0_prenetworkreconfig pve Vri---tz-k 200.00g data vm-100-disk-0 snap_vm-102-disk-2_Neuinstallation pve Vri---tz-k 100.00g data vm-102-disk-2 snap_vm-102-disk-2_Preinstallation pve Vri---tz-k 100.00g data vm-102-disk-2 snap_vm-400-disk-0_prerestore pve Vri---tz-k 1.46t data snap_vm-400-disk-0_preupdate pve Vri---tz-k 1.46t data snap_vm-400-disk-0_preupdate_grooooooss pve Vri---tz-k 1.46t data vm-400-disk-0 snap_vm-400-disk-1_prerestore pve Vri---tz-k 250.00g data snap_vm-400-disk-1_preupdate pve Vri---tz-k 250.00g data snap_vm-400-disk-1_preupdate_grooooooss pve Vri---tz-k 250.00g data vm-400-disk-1 swap pve -wi-ao---- 8.00g vm-100-disk-0 pve Vwi---tz-- 200.00g data vm-100-state-prenetworkreconfig pve Vwi---tz-- <8.49g data vm-102-disk-0 pve Vwi---tz-- 250.00g data vm-102-disk-1 pve Vwi---tz-- 1.46t data vm-102-disk-2 pve Vwi---tz-- 100.00g data vm-102-state-Preinstallation pve Vwi---tz-- <8.49g data vm-103-disk-0 pve Vwi---tz-- 300.00g data vm-104-disk-1 pve Vwi---tz-- 300.00g data vm-104-disk-2 pve Vwi---tz-- 32.00g data vm-105-disk-0 pve Vwi---tz-- 100.00g data base-101-disk-0 vm-106-disk-1 pve Vwi---tz-- 500.00g data vm-108-disk-0 pve Vwi---tz-- 1.46t data vm-109-disk-0 pve Vwi---tz-- 150.00g data vm-112-disk-1 pve Vwi---tz-- 100.00g data vm-115-disk-1 pve Vwi---tz-- 100.00g data vm-400-disk-0 pve Vwi---tz-- 1.46t data snap_vm-400-disk-0_prerestore vm-400-disk-1 pve Vwi---tz-- 250.00g data snap_vm-400-disk-1_prerestore vm-400-state-preupdate pve Vwi---tz-- <32.49g data

 

[soulsurfer42]

...and as an afterthought:

Would it be sensible to move the /root and /swap somewhere outside the LVM?

 

[soulsurfer42]

Anyone? Given that data_tdata activates surely must mean that not all hope is lost. Still any help is appreciated...

 

[soulsurfer42]

Well, thanks for nothing, but to whomever this might happen - this is what worked after an unsuccessful lvchange --repair which provided me with /dev/mapper/pve-data_meta0:

lvcreate -n repaired_01 -L 18G pve thin_dump /dev/mapper/pve-data_meta0 > lvm_meta_dump thin_restore -i lvm_meta_dump -o /dev/mapper/pve-repaired_01 lvconvert --thinpool pve/data --poolmetadata pve/repaired_01 lvchange -a y pve