Context
- Proxmox VE 9.2, rpool in raidz1 on 4 disks (Seagate IronWolf 2TB, ST2000VN004-2E4164)
- After an apt upgrade + reboot, one disk (Z52B3NE5) went from REMOVED to UNAVAIL
- During troubleshooting, several forced power cycles occurred (chained SATA power cable + a separate SATA contact issue on another disk, Z52B3LK1 — both since fixed) which pushed the pool from DEGRADED to FAULTED
Current hardware state (verified stable)
- All 4 disks pass SMART (PASSED)
- Z52B3LK1, Z52B3N1P, Z52B41MT: ONLINE, consistently detected across multiple reboots (verified via lsblk on SystemRescue)
- Z52B3NE5: UNAVAIL, suspected failing disk, currently disconnected
- Power now split across 2 separate cables (no more daisy-chain); SATA contact issue on LK1 fixed by switching port
zpool import
pool: rpool
id: 16558808740828450750
state: FAULTED
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
The pool may be active on another system, but can be imported using
the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:
rpool FAULTED corrupted data
raidz1-0 DEGRADED
ata-ST2000VN004-2E4164_Z52B3LK1-part3 ONLINE
ata-ST2000VN004-2E4164_Z52B3NE5-part3 UNAVAIL
ata-ST2000VN004-2E4164_Z52B3N1P-part3 ONLINE
ata-ST2000VN004-2E4164_Z52B41MT-part3 ONLINE
What has been tried (all fail with "I/O error" or "no such pool or dataset")
- zpool import -f rpool
- zpool import -f -F rpool
- zpool import -f -F -X rpool
- zpool import -f -o readonly=on rpool
- zpool import -f -F -o readonly=on rpool
- zpool import -f -F -X -o readonly=on rpool
- zpool import -f -N -o readonly=on rpool
zdb -l
zdb -l on sda3 (LK1) and sdb3 (N1P) shows the exact same txg: 941712 and identical vdev_tree structure — these two disks appear perfectly consistent with each other. Have not yet been able to check 41MT in the same session (device name changed after reboot).
Update — precise diagnosis via zfs_dbgmsg_enable
After enabling internal debug logs (echo 1 > /sys/module/zfs/parameters/zfs_dbgmsg_enable), attempting import, then cat /proc/spl/kstat/zfs/dbgmsg, the key message that comes up consistently, regardless of the rewind txg attempted (941894 through 941897):
spa_misc.c:417:spa_load_failed(): spa_load(rpool, config untrusted): FAILED: couldn't get 'config' value in MOS directory [error=5]
spa_misc.c:417:spa_load_failed(): spa_load(rpool, config untrusted): FAILED: unable to open rootbp in dsl_pool_init [error=5]
Also tried without success (same errors):
- echo 1 > /sys/module/zfs/parameters/zfs_max_missing_tvds
- echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata
- echo 0 > /sys/module/zfs/parameters/spa_load_verify_data
- zpool import -f -N -o readonly=on rpool
So: the MOS (Meta Object Set) appears unreadable/corrupted in the current configuration, despite 3 ONLINE disks that are consistent with each other (same labels/txg confirmed via zdb -l on sda3 and sdb3).
Question
Is there a manual zdb method to identify/reconstruct the MOS from an earlier, non-corrupted txg, or to pinpoint exactly which block/object is preventing rootbp from opening? Happy to provide any additional output on request (zdb -e -bcsv, zdb -uuu, etc.).
No destructive commands have been run. Thanks in advance for any pointers.
Last edited: