ZFS mirrored rpool disk ID changed, cannot zpool replace

[egeekial]

For my rpool, I have two mirrored NVME SSDs added by-id. However, it seems that one of the ids has changed for some reason, and I am now showing degradation of my rpool:

root@Skywalker:/dev/disk/by-id# zpool status -x
  pool: rpool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: scrub repaired 0B in 0 days 00:04:10 with 0 errors on Sun Apr 12 00:28:15 2020
config:

        NAME                                 STATE     READ WRITE CKSUM
        rpool                                DEGRADED     0     0     0
          mirror-0                           DEGRADED     0     0     0
            10068075249602891436             UNAVAIL      0     0     0  was /dev/disk/by-id/nvme-eui.6479a72a02023637-part3
            nvme-eui.6479a72be0033436-part3  ONLINE       0     0     0

errors: No known data errors

Here are the ids of both disks:

root@Skywalker:/dev/disk/by-id# ls -l | grep nvme
lrwxrwxrwx 1 root root 13 May 16 14:53 nvme-eui.6479a72be0033436 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 May 16 14:53 nvme-eui.6479a72be0033436-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 May 16 14:53 nvme-eui.6479a72be0033436-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 May 16 14:53 nvme-eui.6479a72be0033436-part3 -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 13 May 16 14:53 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046 -> ../../nvme0n1
lrwxrwxrwx 1 root root 15 May 16 14:53 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part1 -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 15 May 16 14:53 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part2 -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 15 May 16 14:53 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3 -> ../../nvme0n1p3

But when I try to zpool replace the disk, I get this error:

root@Skywalker:/dev/disk/by-id# zpool replace rpool /dev/disk/by-id/nvme-eui.6479a72a02023637-part3 /dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3
invalid vdev specification
use '-f' to override the following errors:
/dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3 is part of active pool 'rpool'

I found a similar issue (https://github.com/openzfs/zfs/issues/2076), but trying zpool online command as it suggests results in an error:

root@Skywalker:/dev/disk/by-id# zpool online rpool /dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3
cannot online /dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3: no such device in pool

How can I readd the disk to my rpool? Thanks in advance.

 

[fireon]

You must that like this:

zpool replace <poolname> <old ID> <new ID>

So for you:

zpool replace  rpool 10068075249602891436 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3

So the new/old disk have alread ZFS on, you would like do add "-f" for foce too.

zpool replace  -f rpool 10068075249602891436 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3

But the cleanest way is to format the new disk again, follow my doc's.

Hope this helps

 

[egeekial]

Unfortunately, that gives me a similar error:

# zpool replace -f rpool 10068075249602891436 nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3
invalid vdev specification
the following errors must be manually repaired:
/dev/disk/by-id/nvme-SPCC_M.2_PCIe_SSD_C1D3079C090900503046-part3 is part of active pool 'rpool'

 

[jim.bond.9862]

It says right there. There is a problem with your drive. You need to replace the physical SSD drive and then replace it in the pool. You are trying to add original SSD back when it fails..

Edit: sorry, mistook your new I'd for existing one. Disreguard this post please.

 

[LnxBil]

Removing and Readding could work?