[SOLVED] 1 SSD failed but RAIDZ pool failed

[Curt Hall]

Error I get when attempting an import:

root@proxmox2:~# zpool import -d /dev/disk/by-id
pool: RAIDZ
id: 1648085228702377083
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
config:

RAIDZ UNAVAIL insufficient replicas
wwn-0x5000c5003ea35351 ONLINE
wwn-0x5000c5003ea350f5 ONLINE
wwn-0x5000c5003ea26162 UNAVAIL
wwn-0x5000c5003ea2453a ONLINE
wwn-0x5000c5003ea24522 ONLINE
wwn-0x5000c5003ea28531 ONLINE

3rd disk is the failed one, but I have replaced it with new one but can't get the pool back online.
But notice how all the other disks look fine, so I'm not sure what's going on here.

 

[Dunuin]

You got a 6 disk raid0. So single failed disk means all data is lost...create a new pool using a sriped mirror (raid10) or a raidz1/2/3 (raid5/raid6) so you got some redundancy that your pool can handle 1 to 3 failed disks and then restore data from backups.

 

[Curt Hall]

No "RAIDZ" is same as RAIDZ1, which is 1 parity drive. my zpool should be fine. If not, then in proxmox the definitions are wrong and cost me my data, minus backup recovery.

 

[Dunuin]

RAIDZ UNAVAIL insufficient replicas
wwn-0x5000c5003ea35351 ONLINE
wwn-0x5000c5003ea350f5 ONLINE
wwn-0x5000c5003ea26162 UNAVAIL
wwn-0x5000c5003ea2453a ONLINE
wwn-0x5000c5003ea24522 ONLINE
wwn-0x5000c5003ea28531 ONLINE

According to your output it is a stripe (=raid0) and not a raidz/raidz1. Then there should be a "raidz-0" vdev shown. Like this:

        NAME        STATE
        RAIDZ       UNAVAIL
          raidz-0   UNAVAIL
            wwn-0x5000c5003ea35351  ONLINE
            wwn-0x5000c5003ea350f5  ONLINE
            wwn-0x5000c5003ea26162  UNAVAIL
            wwn-0x5000c5003ea2453a  ONLINE
            wwn-0x5000c5003ea24522  ONLINE
            wwn-0x5000c5003ea28531 ONLINE

 

[Curt Hall]

According to your output it is a stripe (=raid0) and not a raidz/raidz1. Then there should be a "raidz-0" vdev shown. Like this:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
           raidz-0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c4t0d0  ONLINE       0     0     0
            c5t0d0  ONLINE       0     0     0

Well, then where is the RAIDZ-1 in the drop down list in Proxmox as shown in my screenshot?

 

[Dunuin]

Well, then where is the RAIDZ-1 in the drop down list in Proxmox as shown in my screenshot?

Raidz and raidz1 is the same. The point is that your failed pool wasn't created as raidz nor raidz1 but as a "Single Disk" aka stripe aka raid0. Only thing of your pool that is raidz is the name you gave it...so its a raid0 you named "RAIDZ". You probably created the pool using the CLI and wanted to create a raidz but your command was wrong and instead of creating a raidz vdev you created a pool named "RAIDZ" but as striped single disk vdevs...

"zpool create tank raidz /dev/sda ..." will create a pool called tank as a raidz. "zpool create raidz /dev/sda ..." will create a pool called raidz as a raid0.

 

[Curt Hall]

Raidz and raidz1 is the same. The point is that your failed pool wasn't created as raidz nor raidz1 but as a "Single Disk" aka stripe aka raid0. Only thing of your pool that is raidz is the name you gave it...so its a raid0 you named "RAIDZ".

I made the pool from the drop down, so you're telling me that Proxmox goes from a stripe to RAIDZ2 (parity 2 disk), where there is no single disk parity? like hardware RAID5? If so in what universe does that make any sense that Proxmox would offer such a selection? Hopefully an Admin can see this thread and clear this up, its nonsense to me.

 

[Dunuin]

Run a zpool history RAIDZ | grep "zpool create" and you will see as the first entry of the pools log the command it was created with. Then you can verify if it was created as a raid0 or raidz.

 

[Curt Hall]

Run a zpool history RAIDZ | grep "zpool create" and you will see as the first entry of the pools log the command it was created with. Then you can verify if it was created as a raid0 or raidz.

root@proxmox2:~# zpool history RAIDZ
cannot open 'RAIDZ': no such pool

 

[Dunuin]

Ok, then it looks like it needs to be imported to read the log.

 

[Curt Hall]

here's the command in bash history, not zpool history:
zpool create -f -o ashift=12 RAIDZ /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh

 

[Dunuin]

here's the command in bash history, not zpool history:
zpool create -f -o ashift=12 RAIDZ /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh

Yes, that creates a raid0 not a raidz. Raidz would be "zpool create -f -o ashift=12 RAIDZ raidz /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh"

 

[Curt Hall]

ok, somehow through the Proxmox system we can create RAIDZ1 pools as stripes, here's another server setup exactly the same way which looks like RAIDZ1 (parity), but maybe it isn't?:

 

[Dunuin]

ok, somehow through the Proxmox system we can create RAIDZ1 pools as stripes, here's another server setup exactly the same way which looks like RAIDZ1 (parity), but maybe it isn't?:

View attachment 55337

That's a normal single vdev raidz not a stripe. For a striped raidz you need to use the CLI. For example zpool create -f -o ashift=12 RAIDZ raidz /dev/sdc /dev/sdd /dev/sde raidz /dev/sdf /dev/sdg /dev/sdh for two striped 3-disk raidz vdevs. And then add it as a zfs storage with something like pvesm add zfspool RAIDZ --blocksize 32K --content rootdir,images --pool RAIDZ --sparse 1 --mountpoint /RAIDZ.

 

[Curt Hall]

Beats me, all 3 server were created at same time but magically one got stripe. hmmm
oh well, restore from backups it is I guess. thanks for helping.

 

[Dunuin]

Maybe you forgot to change the "Raid level" dropdown from the default "Single Disk" which creates a stripe/raid0 to "raidz" for the third node?