Restore Deleted Files from NFS

[osobh]

Hello,

this morning was horrible. I accidentally deleted a VM that was being served from an NFS share. What advice is there for restoring this VM. FreeNas is exporting this NFS share which is ontop a RAID-Z volume. Any help would be appreciated..

 

[p3x-749]

so you deleted the VM disk with it, right?...and I gather you did NOT run ZFS snapshots on that said Raid-Z, or did you ?

 

[p3x-749]

Don't have snaps?
...here's something to read:

http://mbruning.blogspot.de/2009/12/zfs-raidz-data-walk.html
http://www.osdevcon.org/2008/files/osdevcon2008-max.pdf
http://www.solarisinternals.com/wiki/index.php/ZFS_forensics_scrollback_script
http://serverfault.com/questions/351578/undelete-zfs-file-shared-by-samba

..good luck!

 

[osobh]

Don't have snaps?
...here's something to read:

http://mbruning.blogspot.de/2009/12/zfs-raidz-data-walk.html
http://www.osdevcon.org/2008/files/osdevcon2008-max.pdf
http://www.solarisinternals.com/wiki/index.php/ZFS_forensics_scrollback_script
http://serverfault.com/questions/351578/undelete-zfs-file-shared-by-samba

..good luck!

You are correct in that I did not setup snapshots on this system yet. I will have to backup everything that remains now and then test out these links and see if it's possible to revive this vm.

 

[osobh]

Is it possible to snapshot the volume and then dig thought that? even though I deleted the vm off of it?

 

[p3x-749]

...I don't know...but if you didn't have snaps and did continue to use the volume in the meantime, your data is most likely gone anyways.
There's a hint in the comments in the last link...AFAI understand that ZFS is doing some sort of "silent snaps"...maybe in order to
be able to create a real snap fast....but these will most likely roll over when you use the pool.

 

[osobh]

...I don't know...but if you didn't have snaps and did continue to use the volume in the meantime, your data is most likely gone anyways.
There's a hint in the comments in the last link...AFAI understand that ZFS is doing some sort of "silent snaps"...maybe in order to
be able to create a real snap fast....but these will most likely roll over when you use the pool.

Wonder if I "dd" the zpool dataset onto another single disk if that will capture all the block level data. Any thoughts on doing that?

 

[p3x-749]

Wonder if I "dd" the zpool dataset onto another single disk if that will capture all the block level data. Any thoughts on doing that?

I don't think it'll work out that way.
Your Raid-Z consists of a vdev, build from several physical disks.
You maybe could clone it with dd onto another identical set of disks, outside of ZFS
You could clone a ZFS Volume, since this represents the Blockdevice level of the vdev but I doubt your setup looks like that.

...but if this will copy the "silent snaps", I do not know.
Most likely you can clone the vdev with the same set of spare disks and dd, outside of ZFS, I think.
Then you'll have a spare set to work on or rather to continue business with the cloned FS and work with the original set to undelete your data.

 

[osobh]

I don't think it'll work out that way.
Your Raid-Z consists of a vdev, build from several physical disks.
You maybe could clone it with dd onto another identical set of disks, outside of ZFS
You could clone a ZFS Volume, since this represents the Blockdevice level of the vdev but I doubt your setup looks like that.

...but if this will copy the "silent snaps", I do not know.
Most likely you can clone the vdev with the same set of spare disks and dd, outside of ZFS, I think.
Then you'll have a spare set to work on or rather to continue business with the cloned FS and work with the original set to undelete your data.

That was my initial reaction, to clone this situation somewhere else where I can have the time to explore the innards of this filesystem. I have a feeling that DD would clone all the blocks onto a different disk, but as it stands there are no tools that can identify this type of partition, let alone be able to read all the blocks in this cloned pool. So at least i'll be sitting on the disks or can send them out for data recovery and hope that we can salvage something. In the meantime I have backups from last month that i've rolled back to and now the patching begins. Most of my data is there, thank god for my distributed replicated image store, so all the images are alive just the mysql links that need to rebuilt for all the existing images...anyhow very valuable lesson in Filesystem selection as well as backing up. Had the snaps been working I would have rolled back to the night before, lesson learned.

 

[p3x-749]

I have a feeling that DD would clone all the blocks onto a different disk, but as it stands there are no tools that can identify this type of partition, let alone be able to read all the blocks in this cloned pool. So at least i'll be sitting on the disks or can send them out for data recovery and hope that we can salvage something.

Well, the whole point in this is to clone with DD on raw device level....no partitions and other layers come into play.
You'll probably be needing the exact number of disks, same models and Firmware-version as in your RAID-Z pool to play it safe
You can clone the disks one by one...
Since you cannot risk a change in the FS, you cannot export the pool beforehand.
You could clone the individual disks, put them in another ZFS box and try to import the pool (hmmm...I don't know how you do that without exporting it first).
After you have that cloned pool, you can start working on the forensic part.

In the meantime I have backups from last month that i've rolled back to and now the patching begins. Most of my data is there, thank god for my distributed replicated image store, so all the images are alive just the mysql links that need to rebuilt for all the existing images...anyhow very valuable lesson in Filesystem selection as well as backing up. Had the snaps been working I would have rolled back to the night before, lesson learned.

Yepp...autosnap is your friend...and since ZFS snaps do cost you hardly any space, I'd never go without