ZFS Disk FAULTED after reboot

[Andreas Pflug]

I've been using /dev/disk/by-id/... names.
Both systems are mature production systems, which didn't show the problem until upgrading from 6.8 kernel to 7.0.

 

[Andreas Pflug]

I've been scanning the openzfs github issues a little, and apparently there's a multitude of problems known with imports failing from some mystic reasons. Specifically, in issue https://github.com/openzfs/zfs/issues/10891, still open since 2020, a race condition is suspected with added delay as proposed workaround.
I'm staying with controlled manual import for now.

 

[daanw]

I've been using /dev/disk/by-id/... names.
Both systems are mature production systems, which didn't show the problem until upgrading from 6.8 kernel to 7.0.

Ok good. Because changing device names are known to cause problems on boot.

Specifically, in issue https://github.com/openzfs/zfs/issues/10891, still open since 2020, a race condition is suspected with added delay as proposed workaround.

That race condition would be the result of disabling the deprecated systemd-udev-settle service, still being triggered by the zfs-import-cache service with good reason. Structural chance of zpools failing on boot would be really alarming.

I understand your concerns, problems without clear error messages are the worst, especially regarding large pools of data.

If the problem is timeout/assemble related, the data on the disks should be indifferent and you should for example be able to import the pool on another system. Having this issue with 2 pools (on one system?) is another concern. Can you post a bit more about your setup? Which pool on which controller, controller config, etc.

 

[Andreas Pflug]

I didn't try to import on different systems, just the same problem on two similar systems.

Both systems use megasas controllers with SAS/SCSI backplanes and 12 18TB disks each. Both pools imported with minimum possible disk count automatically, omitting all redundancy (one raid-z1, the other raid-z2). Apparently, ZFS was really eager to start servicing the pool as early as possible...

Since there's only one pool on each system, I don't fear mis-assignment of disks without zpool.cache so I'm probably fine now (well, sort-of).