Hello everybody,
Please I need help !
The LVM-thin metadata corruption occurred after a RAID5 failure involving two failed disks. As a consequence of the RAID degradation, the metadata became corrupted and the thin pool became unavailable.
At this point, we are not looking for the root cause anymore, as it has already been identified.
Our objective is to determine whether there is any possible method to recover the LVM-thin metadata, even partially.
Based on our investigation with thin_explore, we can still access valid mapping trees, internal nodes, leaf nodes, and valid logical-to-physical block mappings.
Therefore, our main interest is not full data recovery support, but whether there is any known method, tool, procedure, or community experience that could allow reconstruction of part of the metadata database from the remaining readable structures.
Even recovering a single VM would be extremely valuable.
LVM-THIN METADATA CORRUPTION INCIDENT
Context :
The LVM-thin pool "pve/data" became unavailable following metadata corruption.
Affected virtual machines:
VM 100
- System disk: 120 GB
- Data disk: ~2 TB
VM 101
- System disk: 500 GB
VM 102
- System disk: 500 GB
The data stored in these virtual machines is considered critical.
====================================================================
Actions Performed
Metadata Backup
A complete copy of the original thin-pool metadata was extracted and saved as:
data_tmeta-original.img
No write operation has ever been performed on this image.
====================================================================
Analysis Using thin-provisioning-tools 1.3.4
The latest official version was compiled and tested:
thin-provisioning-tools 1.3.4
The development tools were also compiled:
thin_explore
thin_stat
====================================================================
thin_check Results
The original metadata image reports:
5300 nodes in data mapping tree contain errors
4817 checksum errors
Thin device 3 is missing root with 1 mappings
Thin device 4:
missing 80977 mappings
expected 429334
Thin device 5:
missing 455422 mappings
expected 1814328
Thin device 6:
missing 51069 mappings
expected 182331
Thin device 7:
missing 1372859 mappings
expected 3900083
====================================================================
thin_repair Results
thin_repair completed successfully:
Exit code = 0
The repaired image also passed thin_check:
Exit code = 0
However, thin_ls on the repaired metadata image only shows:
DEV 1 -> 0 B
DEV 2 -> 0 B
DEV 3 -> 0 B
DEV 4 -> 113 MiB
This indicates that while the repaired metadata is structurally consistent, it appears to have lost the vast majority of mappings.
For this reason, the repaired metadata image was NOT reinjected into production.
====================================================================
Investigation Using thin_explore
The original metadata image was analyzed using:
thin_explore
Device Tree Analysis
The Device Details Tree is readable.
All 7 devices are still present:
Device 1
Device 2
Device 3
Device 4 -> 429334 mappings
Device 5 -> 1814328 mappings
Device 6 -> 182331 mappings
Device 7 -> 3900083 mappings
====================================================================
Mapping Tree Analysis
The main Mapping Tree is also readable.
Observed mapping roots:
Device 4 -> root 216698
Device 5 -> root 216667
Device 6 -> root 216563
Device 7 -> root 216421
====================================================================
Device 7 Exploration
Navigation performed:
216421 INTERNAL
->
69654 INTERNAL
->
26509 LEAF
The leaf node contains valid mappings such as:
25746 x 52 -> 2399192 @ 0
25798 -> 2399244 @ 0
25799 x 13 -> 2399245 @ 0
25812 x 5 -> 2399260 @ 0
...
This demonstrates that:
- The mapping trees still exist.
- Multiple branches remain readable.
- Valid mappings are still present in the original metadata.
- The metadata is damaged but not completely destroyed.
====================================================================
thin_stat Results
thin_stat reports:
checksum errors
and
Index entry points to block (206322) that isn't a bitmap
which suggests corruption affecting part of the metadata space-map structures and mapping trees.
====================================================================
Important Note
The repaired metadata image:
data_tmeta-fixed-1.3.4.img
has NEVER been written back to production.
No commands such as:
dd if=... of=/dev/sdb3
or
dd if=... of=/dev/pve/data_tmeta
have ever been executed.
Production metadata remains untouched.
====================================================================
Assistance Requested
We would like to know:
1. Is there any Proxmox / LVM-thin recovery procedure capable of extracting still-readable mappings directly from:
data_tmeta-original.img
even when the standard thin_repair utility is unable to reconstruct them properly?
2. Is there any internal tool, advanced recovery procedure, or forensic method capable of rebuilding metadata from the still-readable mapping branches observed in:
Device 4
Device 5
Device 6
Device 7
3. Can Proxmox Forum provide an advanced recovery procedure for cases where:
thin_repair produces structurally valid metadata
but discards most of the mappings that still appear accessible through thin_explore?
====================================================================
Available Files
The following files have been preserved and can be provided if required:
data_tmeta-original.img
data_tmeta-before-injection.img
pve-before-repair.cfg
thin-recovery-work.tgz
====================================================================
Summary
The metadata is not completely destroyed.
Using thin_explore, we confirmed that:
- Device records still exist.
- Mapping trees still exist.
- Several internal nodes remain readable.
- Valid leaf mappings are still present.
However, the standard recovery tools (thin_repair and thin_dump --repair) reconstruct only a very small subset of the metadata, resulting in an almost empty recovered image.
We are therefore looking for an advanced recovery method capable of leveraging the still-readable portions of the original metadata image.
Please I need help !
The LVM-thin metadata corruption occurred after a RAID5 failure involving two failed disks. As a consequence of the RAID degradation, the metadata became corrupted and the thin pool became unavailable.
At this point, we are not looking for the root cause anymore, as it has already been identified.
Our objective is to determine whether there is any possible method to recover the LVM-thin metadata, even partially.
Based on our investigation with thin_explore, we can still access valid mapping trees, internal nodes, leaf nodes, and valid logical-to-physical block mappings.
Therefore, our main interest is not full data recovery support, but whether there is any known method, tool, procedure, or community experience that could allow reconstruction of part of the metadata database from the remaining readable structures.
Even recovering a single VM would be extremely valuable.
LVM-THIN METADATA CORRUPTION INCIDENT
Context :
The LVM-thin pool "pve/data" became unavailable following metadata corruption.
Affected virtual machines:
VM 100
- System disk: 120 GB
- Data disk: ~2 TB
VM 101
- System disk: 500 GB
VM 102
- System disk: 500 GB
The data stored in these virtual machines is considered critical.
====================================================================
Actions Performed
Metadata Backup
A complete copy of the original thin-pool metadata was extracted and saved as:
data_tmeta-original.img
No write operation has ever been performed on this image.
====================================================================
Analysis Using thin-provisioning-tools 1.3.4
The latest official version was compiled and tested:
thin-provisioning-tools 1.3.4
The development tools were also compiled:
thin_explore
thin_stat
====================================================================
thin_check Results
The original metadata image reports:
5300 nodes in data mapping tree contain errors
4817 checksum errors
Thin device 3 is missing root with 1 mappings
Thin device 4:
missing 80977 mappings
expected 429334
Thin device 5:
missing 455422 mappings
expected 1814328
Thin device 6:
missing 51069 mappings
expected 182331
Thin device 7:
missing 1372859 mappings
expected 3900083
====================================================================
thin_repair Results
thin_repair completed successfully:
Exit code = 0
The repaired image also passed thin_check:
Exit code = 0
However, thin_ls on the repaired metadata image only shows:
DEV 1 -> 0 B
DEV 2 -> 0 B
DEV 3 -> 0 B
DEV 4 -> 113 MiB
This indicates that while the repaired metadata is structurally consistent, it appears to have lost the vast majority of mappings.
For this reason, the repaired metadata image was NOT reinjected into production.
====================================================================
Investigation Using thin_explore
The original metadata image was analyzed using:
thin_explore
Device Tree Analysis
The Device Details Tree is readable.
All 7 devices are still present:
Device 1
Device 2
Device 3
Device 4 -> 429334 mappings
Device 5 -> 1814328 mappings
Device 6 -> 182331 mappings
Device 7 -> 3900083 mappings
====================================================================
Mapping Tree Analysis
The main Mapping Tree is also readable.
Observed mapping roots:
Device 4 -> root 216698
Device 5 -> root 216667
Device 6 -> root 216563
Device 7 -> root 216421
====================================================================
Device 7 Exploration
Navigation performed:
216421 INTERNAL
->
69654 INTERNAL
->
26509 LEAF
The leaf node contains valid mappings such as:
25746 x 52 -> 2399192 @ 0
25798 -> 2399244 @ 0
25799 x 13 -> 2399245 @ 0
25812 x 5 -> 2399260 @ 0
...
This demonstrates that:
- The mapping trees still exist.
- Multiple branches remain readable.
- Valid mappings are still present in the original metadata.
- The metadata is damaged but not completely destroyed.
====================================================================
thin_stat Results
thin_stat reports:
checksum errors
and
Index entry points to block (206322) that isn't a bitmap
which suggests corruption affecting part of the metadata space-map structures and mapping trees.
====================================================================
Important Note
The repaired metadata image:
data_tmeta-fixed-1.3.4.img
has NEVER been written back to production.
No commands such as:
dd if=... of=/dev/sdb3
or
dd if=... of=/dev/pve/data_tmeta
have ever been executed.
Production metadata remains untouched.
====================================================================
Assistance Requested
We would like to know:
1. Is there any Proxmox / LVM-thin recovery procedure capable of extracting still-readable mappings directly from:
data_tmeta-original.img
even when the standard thin_repair utility is unable to reconstruct them properly?
2. Is there any internal tool, advanced recovery procedure, or forensic method capable of rebuilding metadata from the still-readable mapping branches observed in:
Device 4
Device 5
Device 6
Device 7
3. Can Proxmox Forum provide an advanced recovery procedure for cases where:
thin_repair produces structurally valid metadata
but discards most of the mappings that still appear accessible through thin_explore?
====================================================================
Available Files
The following files have been preserved and can be provided if required:
data_tmeta-original.img
data_tmeta-before-injection.img
pve-before-repair.cfg
thin-recovery-work.tgz
====================================================================
Summary
The metadata is not completely destroyed.
Using thin_explore, we confirmed that:
- Device records still exist.
- Mapping trees still exist.
- Several internal nodes remain readable.
- Valid leaf mappings are still present.
However, the standard recovery tools (thin_repair and thin_dump --repair) reconstruct only a very small subset of the metadata, resulting in an almost empty recovered image.
We are therefore looking for an advanced recovery method capable of leveraging the still-readable portions of the original metadata image.
