LVM Volume Recovery

K

kcactc

New Member
Sep 12, 2026
5
0
1
Need some assistance recovering an LVM (thin) partition that contains my VMs.
TLDR version - An unanticipated reboot of of my PVE server, resulted in a boot to a Win11 boot partition on a drive recently installed in the server. Win11 rewrote the partition tables (uninvited)v on the LVM disk to a 16mb fat partition with the rest of the 256gb drive unallocated.

Steps taken thus far:
1) rebooted to PVE. Drives showed up as offline in PE management portal, and deeper inspection via terminal showed the disk with FAT partition.
2) immediate gddrescue of entire device to an img file on another drive.
3) boot to ubuntu live cd (after proxmox install iso frustrated me a bit on boot to live or rescue env.)
4) ran testdisk and it found an LVM volume. restored that LVM volume but it's strangely (at least to me) just root, var, srv, and swap partitions named after one of the VMs which files on the disk. I have not been able to mount any of those partition to read contents.
5) searched mounted pve root and I do have files in /etc/lvm/backup and /archive.
6) attempts to restore part map have failed with the device/part IDs not found.

Desired outcome:
1) Restore the volume to a readable state so I can copy off the VMs.
-OR-
2) Copy off the vhd of the most valuable VM, meshcentral.

Root cause: A series of unfortunate circumstances that invited the the universe to impose Murphy's Law with Finagle's Corollary of Dynamic Negatives. Expected device behavior coupled with an interim testing environment, interrupted technical process, and errant foot interaction with a powerstrip. I can provide a full explanation as part of closeout if anyone wants it.
Anticipated questions:
- There's no proxmox backup in place (I know)
- there's no other pre-crisis backups.
- It's a house of cards situation: PVE was a proof of concept trial that was hosting two other proof of concept trials as an operational test. What I really care about is recovering the meshcentral poc.

Supposedly I should be able to recreate the partition maps using the lvm backup / archive files. This is based on the Herztner kb article, https://docs.hetzner.com/robot/dedicated-server/troubleshooting/recovery-of-lvm-volumes/
The other complication is that booted into PVE, the affected disk shows up as sdb, but when booted from a live cd it shows up as sda. This is not unexpected hardware behavior, but is a complication.

The archive copy of the vmpool1 .vg file: https://privatebin.net/?277185d366d49912#AWWWEXiKy5Bj5P4ivLhdiE9SfiHpFi5kBMCtpE3L2ECR
The backup folder copy of the vmpool1 .vg file: https://privatebin.net/?bf1027f2a6ad9137#ANg8tjARYx5ZJRw8jyscbsbLa399i1TE9QseSbe8Fjza
Output of testdisk and lsblk: https://privatebin.net/?e7a980272691699f#65K4P3XUtdhsAVVpgapRwd25Csy8ajKn5sXnQKz8wkir
(Let me know if there's a another, preferred, way to share these files.)
 
The good news is that overwriting a(n essential) part of a drive is not specific to Proxmox and various other guides and support forums on the internet might help. LVM is a common technology and if Windows does this automatically then it's probably not an uncommon problem. testdisk seems able to detect the LVM inside the virtual disk(s) of the VM, so you can probably recover that data.
The bad news is that your probably have to pay a data recovery company or find someone knowledgeable with free time (as we are all volunteers on this forum) and who cares more about your data than you apparently did.
 
Thanks for spending the 30 seconds it cost you to post that reply. May it shower upon you all the karma it deserves
.
 
I was hoping that that partition rewrite process using the lvm backup in /etc/lvm would sort it out since there was potentially only a 16mb write to disk. Restore the partition map and get back the mountable partition with hopefully the only first 16mb corrupted. Cie la vie, if not. I was hoping someone had some experience with LVM recovery could giver me some pointers.
 
Really confused by why the lvm partition that testdisk found shows up like the file system of one of the VMs.
 
Because the VM used LVM inside the virtual disk which was places on the LVM of the host. This is why Proxmox shields the (insides of the) virtual disks from LVM, otherwise LVM would get confused. The administration of the outer LVM is now gone but the inner LVM still exists.

There are probably ways to get that data (which was inside the VM) but I am not experience with LVM and data recovery enough. Maybe search for 'LVM data recovery' to find someone who can help or explain it to you in detail? I would expect that LVM recovery is not uncommon, just not very specific to Proxmox, which is why you pobably (also) need to find the experts elsewhere (where they are probably more common).

EDIT: Have you tried copying the detected LVM (whole and with its partition table) to a virtual disk of a VM on another Proxmox? If you manage that and boot that VM with a Linux rescue ISO/USB then you can probably access all files that were inside that VM (or maybe simply boot repair it and have it run as before).
 
Last edited:
kcactc said:
Really confused by why the lvm partition that testdisk found shows up like the file system of one of the VMs.
Click to expand...
testdisk doesnt make distinction between lvm or raw blocks. it looks for file system signatures, and assembled directories/files from available metadata. if that behavior is desirable, run testdisk on the vg/lv instead of the disk but its a very minor distinction.
 
Thank you both.
Alexskysilk - That aligns with what I had read. I was under the assumption that the VM virtual drives are monolithic vhd(ish) files vs partitions on the disk. More study needed on my part.
leesteken - LVM inside LVM aligns with what I was seeing. I ddrescued the original disk to image when it happened and then grabbed an image of the "surfaced" (for lack of the term of art) the MeshCentral partitions into another image when TestDisk restored the detected LVM. None of the copies (original drive, copy of the ddrescue image, or the TestDisk restored MS LVM yielded anything that would mount or could be read.
I pulled the drive and went to work with rstudio on it . Not too heavy a lift since it was a single drive and not really pooled storage. There were files detected of various sizes on both the physical drive and the images. The files were of various sizes and groupings, and file sizes made sense for the LVM-in-a-LVM scenario. Unfortunately in the end, TRIM on the ssd did its job and all files were just zeroes. Total loss, cie la vie.
It's still a little bit of mystery to me as how to reapply the lvm archive/backup and get the drive repartitioned using that but experimentation for another time.

Root cause: This is a proof of concept box we were trialing Proxmox on. It only gets spare parts and that includes drives, so when a NVME drive drive came up for grabs from a decommed user laptop, we stuck it in as better VM pool disk and also to try out virtualizing a windows VM straight from drive as a system recovery option experiment. The Dell box we are using prioritizes the NVME slot as the preferred boot device quite vigorously, so when the unexpected reboot due to power loss (the aforementioned shoe and powerstrip collision) the box rebooted to the windows drive in the NVME slot that was still waiting imaging and wipe. The mesh central vm was a rush request poc with some requirements and constraints that made the lab box (which is beefy other then storage) a great place to run it and that assisted with Proxmox testing as well. The meshcentral did exactly what it was needed for and the requesting team did a fair amount of work on it unbeknownst to us. Enough to make it an appreciable amount of time lost.
I have wiped everything and started fresh now. Boot drive is in zfs raid 1 across the two 256GB drives. The NVME is wiped and repartitioned everything is behaving for the moment. Imaging is running for the box and the vms, and Proxmox BU server is next on the punchlist.
Another silver lining is that Meshcentral agent reinstall is quite forgiving so redeployment of agents was not quite the time suck that was expected.

Thank you both again
 
You must log in or register to reply here.
Top Bottom
Back