Need Help - ZFS repair of Data after Faulted device in Stripe

[Ron14421]

Hello Everyone,
I had a mirrored tank with 2 12TB-HDDs running out of memory.

tank0                                     ONLINE 
    mirror-0                                ONLINE
        ata-TOSHIBA_HDWG21C_Z160A0FYFP8G      ONLINE
        ata-TOSHIBA_MG07ACA12TE_X1P0A0E1F95G  ONLINE

So I decided to put my 2 free 12TB-Drives into the pool.
So I added a mirror with the 2 free drives into my tank0, but because of a read-failure with /dev/disks/by-id.... I used sdc and sdd for a "test".

tank0                                     ONLINE 
    mirror-0                                ONLINE
        ata-TOSHIBA_HDWG21C_Z160A0FYFP8G      ONLINE
        ata-TOSHIBA_MG07ACA12TE_X1P0A0E1F95G  ONLINE
    mirror-1                                ONLINE
       sdc        ONLINE 
       sdd        ONLINE

After that, the drives were running and new available Space was 49% instead of 8%. so far so good.

but because of the possible change in the names of sd* every reboot, I wanted to change the inserted drives from sd* to the correct Drive-ID.

One Drive I could detach/remove, but then the striped mirrors went into a stripe of mirror and a single device

tank0                                     ONLINE 
    mirror-0                                ONLINE
        ata-TOSHIBA_HDWG21C_Z160A0FYFP8G      ONLINE
        ata-TOSHIBA_MG07ACA12TE_X1P0A0E1F95G  ONLINE
    sdd        ONLINE

But I wanted to change the name to the ID, so I tried some further removals of the left "free" disk, but ZPOOL didn't want to detach / remove it.

Because of Proxmox has a Button to swipe disks, I decided to swipe the "free" disk (sdd) and thought I could then remove this disk.
In my opinion, there is no data on the disk, because I had not used it until now.

by the way, the 2 new disks were totally new.

I also tried to put the Disk offline and reboot the Server, but until now the tank wasn't able to get mounted anymore.

ZPOOL import says, that the Pool is unavailable, but lists the devices (the devices containing the data) as Online.

zpool import
   pool: tank0
     id: 12452388303661912470
  state: UNAVAIL
status: One or more devices are faulted.
 action: The pool cannot be imported due to damaged devices or data.
 config:

        tank0                                     UNAVAIL  insufficient replicas
          mirror-0                                ONLINE
            ata-TOSHIBA_HDWG21C_Z160A0FYFP8G      ONLINE
            ata-TOSHIBA_MG07ACA12TE_X1P0A0E1F95G  ONLINE
          ata-TOSHIBA_HDWG21C_Z160A0F8FP8G        FAULTED  corrupted data

zpool import tank0
cannot import 'tank0': one or more devices is currently unavailable

I also tried to create a new tank with only one of the data-containing mirrored disks, but that pool showed 99% free space.
I never touched the zfs command.

Do you know any possibility to get the Data from the disks?

Thank you very much for your Help

 

[Dunuin]

Thats not how raid works. If you remove/loose both disks of a mirror all data of that pool is gone. You can't just remove both disks of a mirror. Data is spread across all 4 disks after making a mirror to a striped mirror.

What you could have done is removing one disk from a mirror, wiping that removed disk and adding it with its /dev/disk/by-id. Then let the pool resilver. Then remove the other disk of that mirror, wipe it, resilver and add it again as /dev/disk/by-id. That way no data would have been lost.

And it wouldn't have been a problem when sdc or sdd changed names. ZFS doesn't care about it after adding the disks, as ZFS reads the metadata from the disks themself so it doesn't matter. Even if sdd would now be sde, ZFS wouldn`t use sdd. It would read all disks, check the metadata and see that sde was added as sdd so it would use sde and not sdd. But using /dev/disk/by-id would still be recommended. That way you can't add wrong disks by accident when modifying your pool layout and its easier to find out what which disk to replace when it fails.

And by the way...a ZFS pool should always have 20% of free capacity or it will become slow. So just expanding it when only 8% is left is very late.