Strange ZFS behavior

[IsThisThingOn]

Something pretty strange happend again to my ZFS pool today, but it is probably best to start from the beginning.

For multiple years I had a WD Red and a Kingston DC NMVE as mirrors.
Few months ago I got a email that the Kingston drive was gone.
I did not think much of it, ordered a 990 Pro and replaced it.
I was a little bit confused on how the Kingston seems to work fine on my NVME to USB-C adapter. No SMART errors.
Anyway, since it was too late now anyway, the Kingston went into the drawer.

Yesterday I installed some updates, rebooted and went to bed.
Probably at the same time, I again got a email.
The 990 Pro is also gone. I immediatly suspected the Mobo to have some kind of defect.
But when I digged a little bit deeper, I found something is odd.

My WD is nvme0n1, with p1 as BIOS, p2 as EFI and p3 as ZFS.
Same is true for my Samsung, which is nmve1n1.

But when I go to the pool status, I can see that the missing Samsung has the Message: "was /dev/nvme1n1p2"

Wait a second! Now my Samsung is /dev/nvme1n1p3, but in the ZFS pool it was /dev/nvme1n1p2?
How could that happen?

 

[IsThisThingOn]

Uff replacing the disk was a little bit of a challenge, but should work for now.

zpool detach rpool nvme1n1p3

Deleted the disk in the webGUI

sgdisk /dev/nvme0n1 -R /dev/nvme1n1
sgdisk --randomize-guids /dev/nvme1n1
proxmox-boot-tool format /dev/nvme1n1p2
proxmox-boot-tool init /dev/nvme1n1p2
zpool attach rpool /dev/disk/by-id/nvme-eui.e8238fa6bf530001001b448b4e348aea-part3 /dev/disk/by-id/nvme-eui.00253842514132b5-part3

Where I had problems:
- Overlooked that I at first need sgdisk. That way you copy the partitions over.
- After that, since it is a boot pool, I would make it bootable again with proxmox-boot-tool. There I need parition 2! Since that is where EFI is located.
- Only in the last step, I mirror the ZFS pool. I was under the wrong impression that the whole two disks are mirrored, when in fact it is only the ZFS data partition. That is partition 3!

The official doc only states stuff like <device_id> which was not that helpful if you didn't understand that not the whole disk is mirrored