ZFS Disk FAULTED after reboot

Andreas Pflug · Jul 16, 2026

I recently upgraded two machines with big ZFS pools: one to PVE9.2.4, the other to PBS4.2.2, so both machines have Kernel 7.0 and ZFS 2.4.3 now.
One pool is raid-z1 (12x18TB), had one drive faulted, the other raid-z2 (12x18TB), with two drives faulted after reboot. zpool-online didn't help, I had to zpool-labelclear and zpool-replace the disks, with the consequence of several days of resilver.
With previous system versions, I can't remember having such problems, so I suspect that newer kernels/zfs might have different timings, with disks not ready in-time from the SAS2008/3008 adapters. storcli will show all disk as good, smartctl didn't show any problems on the faulted disks.

Are there any best-practices to avoid such reboot trouble?

Regards,
Andreas

news · Jul 16, 2026

I'm interested, please show the results of:

Code:

zpool status <pool-name>
zpool list -v <pool-name>

Andreas Pflug · Jul 16, 2026

zpool status -L
pool: ironpool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Jul 15 11:31:13 2026
30.8T / 115T scanned at 366M/s, 30.2T / 115T issued at 359M/s
2.55T resilvered, 26.41% done, 2 days 20:20:43 to go
config:

NAME STATE READ WRITE CKSUM
ironpool DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
sda ONLINE 0 0 0
sdi ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdk ONLINE 0 0 0
sdc ONLINE 0 0 0
replacing-6 DEGRADED 0 0 0
9235897770678688098 OFFLINE 0 0 0 was /dev/disk/by-id/scsi-SATA_ST18000NM000J-2T_REDACTED-part1
sdl ONLINE 0 0 0 (resilvering)
sdb ONLINE 0 0 0
sdd ONLINE 0 0 0
sdh ONLINE 0 0 0
sdg ONLINE 0 0 0
sdj ONLINE 0 0 0
logs
mirror-1 ONLINE 0 0 0
nvme0n1p3 ONLINE 0 0 0
nvme1n1p3 ONLINE 0 0 0

errors: No known data errors

zpool list -v

NAME                                                    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT

ironpool                                                196T   115T  81.6T        -         -    48%    58%  1.00x  DEGRADED  -

  raidz1-0                                              196T   115T  81.6T        -         -    48%  58.4%      -  DEGRADED        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    replacing-6                                            -      -      -        -         -      -      -      -  DEGRADED        -

      9235897770678688098                                  -      -      -        -         -      -      -      -   OFFLINE        -

    wwn-0x5000c500redacted.                            16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

    wwn-0x5000c500redacted                             16.4T      -      -        -         -      -      -      -    ONLINE        -

logs                                                       -      -      -        -         -      -      -      -         -        -

  mirror-1                                              149G  86.8M   149G        -         -     0%  0.05%      -    ONLINE        -

    nvme-INTEL_SSDPED1D480GA_PHMB75130REDACTEDGN-part3   150G      -      -        -         -      -      -      -    ONLINE        -

    nvme-INTEL_SSDPED1D480GA_PHMB75130REDACTEDGN-part3   150G      -      -        -         -      -      -      -    ONLINE        -

Andreas Pflug · Sunday at 14:02

No more hints here...
I re-configured ZFS to not automatically import the pool (`zpool set cachefile=none <poolname>`), so I have to import the pool manually after each reboot. So far, this worked fine.
So apparently there IS a timing problem.

xrobau · Sunday at 14:13

> 2 days 20:20:43 to go

Wait the 2 days, then you should be fine.

Andreas Pflug · Sunday at 14:19

I don't intend to have delicate manual intervention and 2-3 days of non-redundancy after each reboot.

xrobau · Sunday at 14:22

Andreas Pflug said:
I don't intend to have delicate manual intervention and 2-3 days of non-redundancy after each reboot.

As someone who has exactly the same setup, ZFS has determined your drives have failed.

Wait for the scrub to finish.

After the scrub has finished, run multiple scrubs. If you discover things are failing, the most likely failure cause is your HBA, but you will need to investigate this for yourself.

Don't disregard the zfs errors. They're telling you what the problem is.

Andreas Pflug · Sunday at 14:25

Please read my first thread.
ZFS did nothing to automatically fix the pool, instead I had to labelclear the disks and add them as fresh devices.

xrobau · Sunday at 14:31

Andreas Pflug said:
Please read my first thread.
ZFS did nothing to automatically fix the pool, instead I had to labelclear the disks and add them as fresh devices.

I did skip over that, because I thought it was just a language issue. You reused a drive that ZFS had already deemed unreliable?

I don't think that was a wise decision. 'zpool status' and 'dmesg' would have shown you what the actual problem was.

Andreas Pflug · Sunday at 14:39

Please read my first thread.
The devices are fully functional without any defects or degraded metrics reported by SMART, no anomalies reported in kern.log.

xrobau · Sunday at 14:39

Andreas Pflug said:
Please read my first thread.
The devices are fully functional without any defects or degraded metrics reported by SMART, no anomalies reported in kern.log.

Except ZFS is saying it's not.

Please read MY response.

Andreas Pflug · Sunday at 14:43

ZFS doesn't give any hint what it doesn't like. It has marked fully functional random disks for unknown reasons.
Since the problems are gone (with the very same pool) after disabling auto-import, the picture is quite clear.

xrobau · Sunday at 14:45

Andreas Pflug said:
ZFS doesn't give any hint what it doesn't like. It has marked fully functional random disks for unknown reasons.
Since the problems are gone (with the very same pool) after disabling auto-import, the picture is quite clear.

I assure you, it is. It will be in dmesg.

As I said, wait for the resilver to complete. Then reun multiple resilvers.

Andreas Pflug · Sunday at 14:49

AGAIN: there was nothing in dmesg/kern.log about disk problems, nor about zfs (except from taint msg).

xrobau · Sunday at 14:51

Andreas Pflug said:
AGAIN: there was nothing in dmesg/kern.log about disk problems, nor about zfs (except from taint msg).

*Shrug* I dunno what to say here.

ZFS just does not randomly mark a drive as bad. There is always a reason.

Citation: Worked for Sun back in the Solaris 2.4 days when ZFS was being created. I use it everywhere.

Andreas Pflug · Sunday at 14:59

I'd love to know the reason, but there was no indication anywhere, just "enabled, but still faulty". Since this happened on megasas/IDE drive pools only, not on NVME pools, this appears as some weird timing problem to me.
If there were problems on one of those 24 enterprise grade not-too-old-not-too-new harddisks, they'd probably be noticed in daily workload usage, not on the distinct import moment on three of them.

xrobau · Sunday at 15:01

Andreas Pflug said:
I'd love to know the reason, but there was no indication anywhere, just "enabled, but still faulty". Since this happened on megasas/IDE drive pools only, not on NVME pools, this appears as some weird timing problem to me.
If there were problems on one of those 24 enterprise grade not-too-old-not-too-new harddisks, they'd probably be noticed in daily workload usage, not on the distinct import moment on three of them.

> As I said, wait for the resilver to complete. Then reun multiple resilvers.

Andreas Pflug · Sunday at 15:05

Repeating that hint won't help. The problem was triggered by reboot, not by data traffic. I'm not inclined to provoke that again.

daanw · Sunday at 15:15

Andreas Pflug said:
Are there any best-practices to avoid such reboot trouble?

Was the pool using device names (i.e. /dev/sdX) or the preferred persistent device paths (/dev/disk/by-id) at time of the failure?

UdoB · Sunday at 15:33

Andreas Pflug said:
The problem was triggered by reboot, not by data traffic. I'm not inclined to provoke that again.

If you don't trust the system to be able to reboot, you cannot trust it at all!

It is a pure test-system with only test-data on it, right?

ZFS Disk FAULTED after reboot

Well-Known Member

Famous Member

Well-Known Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Member

Well-Known Member

Active Member

Distinguished Member

We value your privacy