Resilvered for no obvious reason

[Kodey]

My server hangs when this keeps happening (maybe 3 times over 6 months):

ZFS has finished a resilver:

   eid: 9
 class: resilver_finish
  host: pmhost
  time: 2022-09-11 01:18:07+1000
  pool: rpool
 state: ONLINE
  scan: resilvered 424K in 00:00:00 with 0 errors on Sun Sep 11 01:18:07 2022
config:

    NAME                               STATE     READ WRITE CKSUM
    rpool                              ONLINE       0     0     0
      nvme-eui.6479a74210200d1e-part3  ONLINE       0     0     0
      nvme-eui.6479a747a0201215-part3  ONLINE       0     0     0

errors: No known data errors

zpool status has no clues. How can I find the cause?

 

[LnxBil]

My server hangs when this keeps happening (maybe 3 times over 6 months):

What type of NVMe is this?

How can I find the cause?

I assume just silent data corruption. I've seen this also in old (>5y) disk arrays in about the same byte region as yours show. In a disk setup this can also be caused by hba/controller, cables of power problems, but with NVMe, there is no such error source (AFAIK) involved.

 

[Kodey]

What type of NVMe is this?

nvme GIGABYTE_GP-ASM2NE6100TTTD

In a disk setup this can also be caused by hba/controller, cables of power problems, but with NVMe, there is no such error source (AFAIK) involved.

Maybe, From memory, this MB has a raid controller which I believe needs to be enabled at boot time since this zpool contains the boot partition.

I get this probably unrelated log message which happens every boot:

Sep 11 01:18:52 pmhost smartd[40139]: Device: /dev/nvme0, number of Error Log entries increased from 262 to 263
Sep 11 01:18:52 pmhost smartd[40139]: Device: /dev/nvme1, number of Error Log entries increased from 359 to 361
Sep 11 01:18:52 pmhost smartd[40139]: Device: /dev/nvme2, number of Error Log entries increased from 262 to 263

The pool that was resilvered is on device nvme0 and nvme2. The other, a nvme Samsung_SSD_950_PRO_512GB is being used as a cache on a ata pool