ZFS: recover filesystem

[dejhost]

Hi guys,

I've a proxmox server with a ZFS Raid 1 pool (2x 18TB WD disks). I inteded to switch to CEPH, so I created an image of the 2 existing vm-disks, and stored the image files on a second proxmox server.

Since the image proved to be fine, I destroyed the ZFS and initialized CEPH. Adding OSD overwrote about 2GB of data.

This is when my backup-images were rendered useless by a bug (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-0847). So I deactevated CEPH at once.

Do you have a recommandation on how to proceed?
Should I use Photorec to attempt to recover individual files? I would rather recover the original ZFS filesystem - or as much as possible.

Thank you very much for helping.

 

[LnxBil]

This is when my backup-images were rendered useless by a bug (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-0847).

Why did that bug render your PVE backup useless? I thought you tested it, so a once working backup should stay that way.

 

[dejhost]

When testing the decompression of the images, they were fine. But piping that output to the original server... that's when things went wrong.

 

[dejhost]

Am I posting in the wrong forum? Could you point me towards a forum where I might geht help with this?

 

[Neobin]

Reason could be, because you are giving basically no information at all.

For example:

• What did you use to create the backup-images aka what format are they?
• What exactly went wrong with restoring them? Any error messages?
• How did you try restoring them?
• How is the mentioned bug in your opinion involved in the problem?

 

[LnxBil]

What @Neobin said.

and

But piping that output to the original server... that's when things went wrong.

Then don't pipe them.

 

[Dunuin]

Then don't pipe them.

For me it sounds like he copied the VMs from ZFS to CEPH so he got corrupted VMs on CEPH but he already destroyed the ZFS pool so the healthy VMs don't exist anymore. And he wants to know how to rescue data stored on an already destroyed pool. So basically, how to undo a "zpool destroy"?

 

[LnxBil]

The only chance there is, if zpool import sees the already destroyed pool, otherwise maybe a data recovery specialist may be able to help.
I do not know if there are file recovery tools available for ZFS.

I still don't believe that this was the aforementioned bug.

 

[dejhost]

Hi,
Thank you for diving into this.
This is the command used to roll back the images to the newly created CEPH disk.

pv <zfs-block-device> | zstd - --stdout --adapt -z | ssh <ceph-host> dd bs=16M of=/mnt/rbd_backups/<IMAGENAME>.img.zst

It seems, we missed some """ - so source and destiny were identical:


So it wasn't the mentioned bug afterall.

 

[dejhost]

Just making sure, I understand your comments:

1. The damaged zpool can probably not get recovered.
2. Only chance is to send the damaged images to a specialist.

Is there a specific (affordable) service supplier, that you can recommend?

Thank you.

 

[LnxBil]

The damaged zpool can probably not get recovered.

You stated that you've overwritten parts of the disk, so there is nothing to recover there, maybe other parts can be. ZFS is not like any other filesystem, you will not easily be able to extract huge junks of data while going over the disk. Yes, there may be parts up to 128K (recordsize), but it's not likely that you will get all. First, I'd make hard (dd) copies of your disk(s) to another disk to play around, that's always very useful.

 

[dejhost]

You stated that you've overwritten parts of the disk, so there is nothing to recover there, maybe other parts can be. ZFS is not like any other filesystem, you will not easily be able to extract huge junks of data while going over the disk. Yes, there may be parts up to 128K (recordsize), but it's not likely that you will get all. First, I'd make hard (dd) copies of your disk(s) to another disk to play around, that's always very useful.

Thank you for the clarification and the support.