ZFS Disk FAULTED after reboot

[Andreas Pflug]

I recently upgraded two machines with big ZFS pools: one to PVE9.2.4, the other to PBS4.2.2, so both machines have Kernel 7.0 and ZFS 2.4.3 now.
One pool is raid-z1 (12x18TB), had one drive faulted, the other raid-z2 (12x18TB), with two drives faulted after reboot. zpool-online didn't help, I had to zpool-labelclear and zpool-replace the disks, with the consequence of several days of resilver.
With previous system versions, I can't remember having such problems, so I suspect that newer kernels/zfs might have different timings, with disks not ready in-time from the SAS2008/3008 adapters. storcli will show all disk as good, smartctl didn't show any problems on the faulted disks.

Are there any best-practices to avoid such reboot trouble?

Regards,
Andreas

 

[news]

I'm interested, please show the results of:

zpool status <pool-name>
zpool list -v <pool-name>

 

[Andreas Pflug]

zpool status -L
pool: ironpool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 15 11:31:13 2026
30.8T / 115T scanned at 366M/s, 30.2T / 115T issued at 359M/s
2.55T resilvered, 26.41% done, 2 days 20:20:43 to go
config:

NAME STATE READ WRITE CKSUM
ironpool DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sda ONLINE 0 0 0
sdi ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdk ONLINE 0 0 0
sdc ONLINE 0 0 0
replacing-6 DEGRADED 0 0 0
9235897770678688098 OFFLINE 0 0 0 was /dev/disk/by-id/scsi-SATA_ST18000NM000J-2T_REDACTED-part1
sdl ONLINE 0 0 0 (resilvering)
sdb ONLINE 0 0 0
sdd ONLINE 0 0 0
sdh ONLINE 0 0 0
sdg ONLINE 0 0 0
sdj ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
nvme1n1p3 ONLINE 0 0 0

errors: No known data errors

zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ironpool 196T 115T 81.6T - - 48% 58% 1.00x DEGRADED -
raidz1-0 196T 115T 81.6T - - 48% 58.4% - DEGRADED -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
replacing-6 - - - - - - - - DEGRADED -
9235897770678688098 - - - - - - - - OFFLINE -
wwn-0x5000c500redacted. 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
wwn-0x5000c500redacted 16.4T - - - - - - - ONLINE -
logs - - - - - - - - - -
mirror-1 149G 86.8M 149G - - 0% 0.05% - ONLINE -
nvme-INTEL_SSDPED1D480GA_PHMB75130REDACTEDGN-part3 150G - - - - - - - ONLINE -
nvme-INTEL_SSDPED1D480GA_PHMB75130REDACTEDGN-part3 150G - - - - - - - ONLINE -