Hi! First, a quick disclosure. I'm one of the developers at Hetman Software, a Ukrainian company that has been developing commercial data recovery software for over 15 years.
I hope this doesn't come across as self-promotion. I'd like to propose a technical experiment, get feedback from experienced Proxmox administrators, and share what we find, including any failures or limitations.
The idea is to build a Proxmox VE test environment, deliberately break something, and investigate what can actually be recovered.
Here's the initial setup I'm considering:
I'd like the community to suggest what we should break.
A few possible scenarios:
Another question is how we should perform the recovery analysis.
I see two possible approaches:
My background is mainly in low-level storage analysis, filesystem internals, RAID reconstruction, and data recovery. I don't have the same experience administering large production Proxmox environments with many users and virtual machines. So if some of my proposed configurations seem unusual or unrealistic from an administrator's perspective, please point that out. That's exactly the kind of feedback I'm looking for.
What I can contribute is a detailed analysis of what actually gets written to disk during different operations, what metadata and file contents remain afterward, and whether software-based recovery is possible.
I'll document the test environment, recovery process, SHA-256 verification results, and the reasons why recovery succeeds or fails.
We can start with one experiment chosen by the community, or run a series of different scenarios if there's enough interest.
I'd appreciate your suggestions, especially from anyone who has dealt with unusual Proxmox storage failures in real production environments. What would you like us to test?
I hope this doesn't come across as self-promotion. I'd like to propose a technical experiment, get feedback from experienced Proxmox administrators, and share what we find, including any failures or limitations.
The idea is to build a Proxmox VE test environment, deliberately break something, and investigate what can actually be recovered.
Here's the initial setup I'm considering:
- ZFS: A ZFS pool configured as either RAIDZ1 or dRAID1, depending on the community's preference. We'll also use native ZFS encryption, compression, and snapshots.
- LVM: A dedicated disk with thick-provisioned logical volumes.
- LVM-Thin: Another disk with a thin pool and snapshots.
- Directory storage: Ext4 and XFS for file-based virtual disks.
- Virtual machines: Several Windows and Linux VMs using different storage configurations, including RAW, QCOW2, ZVOLs, and thick/thin LVM volumes. We could also include VMDK images.
I'd like the community to suggest what we should break.
A few possible scenarios:
- Accidentally delete one or more VMs, including their virtual disks.
- Delete a ZFS dataset or destroy an entire pool.
- Simulate disk failures in RAIDZ1 or dRAID1, staying within their parity tolerance.
- Delete LVM volumes or damage LVM-Thin metadata.
- Delete snapshots, or combine several destructive operations into a more complicated failure scenario.
Another question is how we should perform the recovery analysis.
I see two possible approaches:
- Run our recovery software inside a dedicated VM, giving it direct, strictly read-only access to the physical disks while keeping the Proxmox host running.
- Shut down the server, connect the disks to another computer, and perform the analysis offline.
My background is mainly in low-level storage analysis, filesystem internals, RAID reconstruction, and data recovery. I don't have the same experience administering large production Proxmox environments with many users and virtual machines. So if some of my proposed configurations seem unusual or unrealistic from an administrator's perspective, please point that out. That's exactly the kind of feedback I'm looking for.
What I can contribute is a detailed analysis of what actually gets written to disk during different operations, what metadata and file contents remain afterward, and whether software-based recovery is possible.
I'll document the test environment, recovery process, SHA-256 verification results, and the reasons why recovery succeeds or fails.
We can start with one experiment chosen by the community, or run a series of different scenarios if there's enough interest.
I'd appreciate your suggestions, especially from anyone who has dealt with unusual Proxmox storage failures in real production environments. What would you like us to test?
