LVM Volume Recovery

[kcactc]

Need some assistance recovering an LVM (thin) partition that contains my VMs.
TLDR version - An unanticipated reboot of of my PVE server, resulted in a boot to a Win11 boot partition on a drive recently installed in the server. Win11 rewrote the partition tables (uninvited)v on the LVM disk to a 16mb fat partition with the rest of the 256gb drive unallocated.

Steps taken thus far:
1) rebooted to PVE. Drives showed up as offline in PE management portal, and deeper inspection via terminal showed the disk with FAT partition.
2) immediate gddrescue of entire device to an img file on another drive.
3) boot to ubuntu live cd (after proxmox install iso frustrated me a bit on boot to live or rescue env.)
4) ran testdisk and it found an LVM volume. restored that LVM volume but it's strangely (at least to me) just root, var, srv, and swap partitions named after one of the VMs which files on the disk. I have not been able to mount any of those partition to read contents.
5) searched mounted pve root and I do have files in /etc/lvm/backup and /archive.
6) attempts to restore part map have failed with the device/part IDs not found.

Desired outcome:
1) Restore the volume to a readable state so I can copy off the VMs.
-OR-
2) Copy off the vhd of the most valuable VM, meshcentral.

Root cause: A series of unfortunate circumstances that invited the the universe to impose Murphy's Law with Finagle's Corollary of Dynamic Negatives. Expected device behavior coupled with an interim testing environment, interrupted technical process, and errant foot interaction with a powerstrip. I can provide a full explanation as part of closeout if anyone wants it.
Anticipated questions:
- There's no proxmox backup in place (I know)
- there's no other pre-crisis backups.
- It's a house of cards situation: PVE was a proof of concept trial that was hosting two other proof of concept trials as an operational test. What I really care about is recovering the meshcentral poc.

Supposedly I should be able to recreate the partition maps using the lvm backup / archive files. This is based on the Herztner kb article, https://docs.hetzner.com/robot/dedicated-server/troubleshooting/recovery-of-lvm-volumes/
The other complication is that booted into PVE, the affected disk shows up as sdb, but when booted from a live cd it shows up as sda. This is not unexpected hardware behavior, but is a complication.

The archive copy of the vmpool1 .vg file: https://privatebin.net/?277185d366d49912#AWWWEXiKy5Bj5P4ivLhdiE9SfiHpFi5kBMCtpE3L2ECR
The backup folder copy of the vmpool1 .vg file: https://privatebin.net/?bf1027f2a6ad9137#ANg8tjARYx5ZJRw8jyscbsbLa399i1TE9QseSbe8Fjza
Output of testdisk and lsblk: https://privatebin.net/?e7a980272691699f#65K4P3XUtdhsAVVpgapRwd25Csy8ajKn5sXnQKz8wkir
(Let me know if there's a another, preferred, way to share these files.)

 

[leesteken]

The good news is that overwriting a(n essential) part of a drive is not specific to Proxmox and various other guides and support forums on the internet might help. LVM is a common technology and if Windows does this automatically then it's probably not an uncommon problem. testdisk seems able to detect the LVM inside the virtual disk(s) of the VM, so you can probably recover that data.
The bad news is that your probably have to pay a data recovery company or find someone knowledgeable with free time (as we are all volunteers on this forum) and who cares more about your data than you apparently did.

 

[kcactc]

Thanks for spending the 30 seconds it cost you to post that reply. May it shower upon you all the karma it deserves
.

 

[alexskysilk]

If testdisk isnt recovering it, its gone. I suspect you already know this.

 

[kcactc]

I was hoping that that partition rewrite process using the lvm backup in /etc/lvm would sort it out since there was potentially only a 16mb write to disk. Restore the partition map and get back the mountable partition with hopefully the only first 16mb corrupted. Cie la vie, if not. I was hoping someone had some experience with LVM recovery could giver me some pointers.

 

[kcactc]

Really confused by why the lvm partition that testdisk found shows up like the file system of one of the VMs.

 

[leesteken]

Because the VM used LVM inside the virtual disk which was places on the LVM of the host. This is why Proxmox shields the (insides of the) virtual disks from LVM, otherwise LVM would get confused. The administration of the outer LVM is now gone but the inner LVM still exists.

There are probably ways to get that data (which was inside the VM) but I am not experience with LVM and data recovery enough. Maybe search for 'LVM data recovery' to find someone who can help or explain it to you in detail? I would expect that LVM recovery is not uncommon, just not very specific to Proxmox, which is why you pobably need to find the experts elsewhere.