ZFS drives FAULTED 'was /dev/sd*'

[mhaluska]

Hi,

I created zpool in PVE webgui, didn't realize it can be problem later, because it used /dev/sd* instead of /dev/disk/by-id/*. Now after reboot my RAID-Z2 is DEGRADED.

# zpool status -x
  pool: dpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0B in 0 days 00:02:58 with 0 errors on Thu Dec 19 20:44:59 2019
config:

        NAME                      STATE     READ WRITE CKSUM
        dpool                     DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            sdc                   ONLINE       0     0     0
            sdd                   ONLINE       0     0     0
            sde                   ONLINE       0     0     0
            14009361960905333539  FAULTED      0     0     0  was /dev/sdf1
            9903023571805836418   FAULTED      0     0     0  was /dev/sdg1
            sdh                   ONLINE       0     0     0

errors: No known data errors

Is there some quick way how to fix this? I want to avoid backup/restore if possible.

 

[fabian]

that should not matter if the disks are actually there.. you can also try exporting and importing again with '-d /dev/diks/by-id' added to the import command.

 

[mhaluska]

This didn't help, it was reimported with same current layout where sdf and sdg has been switched:

# zpool status -x
  pool: dpool
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
        repaired.
  scan: scrub repaired 0B in 0 days 00:02:58 with 0 errors on Thu Dec 19 20:44:59 2019
config:

        NAME                        STATE     READ WRITE CKSUM
        dpool                       DEGRADED     0     0     0
          raidz2-0                  DEGRADED     0     0     0
            wwn-0x55cd2e4150b64b2c  ONLINE       0     0     0
            wwn-0x55cd2e4151653a6d  ONLINE       0     0     0
            wwn-0x55cd2e41515a06eb  ONLINE       0     0     0
            14009361960905333539    FAULTED      0     0     0  was /dev/disk/by-id/wwn-0x55cd2e4150b65e5b-part1
            9903023571805836418     FAULTED      0     0     0  was /dev/disk/by-id/wwn-0x55cd2e41517ed54b-part1
            wwn-0x55cd2e41517c891e  ONLINE       0     0     0

 

[mhaluska]

And ZFS still knows thos devices, they've same id:

# zpool replace -f dpool 14009361960905333539 /dev/disk/by-id/wwn-0x55cd2e4150b65e5b
invalid vdev specification
the following errors must be manually repaired:
/dev/disk/by-id/wwn-0x55cd2e4150b65e5b-part1 is part of active pool 'dpool'

 

[fabian]

can you try 'zpool clear' or 'zpool online' ?

 

[mhaluska]

Hey, I wasn't to patient, so I already destroyed the pool. This machine is not yet used, so I just did backup, now restoring VMs.
I just wanted to try fix it, if something similar will happen when it's used. Anyway now pool is configured using by-id, so this problem should not happen again.
Thanks for help.