Kernel Panic on reboot from ZFS

[daalinaar]

I'm having an issue and can't get my server to boot back to the prompt. Any reboot gives the error

[31.621178] PANIC: zfs: rt={spa=(zfspoolname) vdev_guid=1903510110132383870 ms_id=471 ms_allocatable}: removing nonexistent segment from range tree (offset=76c079f6000 size =300000)

I have tried booting into recovery which works fine, I can view the 3 different zfs pools I have, and none of them show errors and all show online. Could someone offer any insight on this?

 

[cwt]

Looks like a corrupted ZFS space map.

Try in recovery mode:

zpool import
zpool import -N -o readonly=on zfspoolname
zpool status -v
zpool events -v
dmesg -T | grep -iE 'zfs|nvme|ata|error|timeout|reset|checksum|panic'

If read-only works you should immediately backup your data with zfs send/receive on a new pool.

After that you can try:

echo 1 > /sys/module/zfs/parameters/zfs_recover
zpool import -N -f zfspoolname
zpool scrub zfspoolname

A hardware check (SMART, RAM, cable, etc.) is also advisable.

 

[daalinaar]

Thanks for your reply! I went through those steps, none of the commands hung so that was a good sign, no errors on the pool and it was online. I'm now running the scrub. It's about 30TB of data so I'll report back once that is done.

 

[daalinaar]

Looks like a corrupted ZFS space map.

Try in recovery mode:

zpool import
zpool import -N -o readonly=on zfspoolname
zpool status -v
zpool events -v
dmesg -T | grep -iE 'zfs|nvme|ata|error|timeout|reset|checksum|panic'

If read-only works you should immediately backup your data with zfs send/receive on a new pool.

After that you can try:

echo 1 > /sys/module/zfs/parameters/zfs_recover
zpool import -N -f zfspoolname
zpool scrub zfspoolname

A hardware check (SMART, RAM, cable, etc.) is also advisable.

Alright I did the scrub, it didn't find anything but a single error, I removed that file, tried to reboot and the same error occurred when trying to boot. It's like 30TB of stuff so I don't have anywhere to move it to. Anyway to fix this specific error?

 

[cwt]

A scrub only verifies data integrity and checksums. It does not repair corrupted allocation metadata, metaslabs, space maps, or range trees.

The panic you’re seeing (“removing nonexistent segment from range tree”) suggests an inconsistency in ZFS’ internal allocation structures rather than a problem with a specific file. The file error found during the scrub may be completely unrelated.

The fact that recovery mode still works is actually encouraging. Before attempting any further writes, I would try importing the pool read-only:

zpool import -N -o readonly=on

If that succeeds, run:

zpool status -v
zpool events -v

and note your exact OpenZFS version.

I would avoid deleting snapshots, running another scrub, resilvering, or making any other changes until you understand what’s causing the panic. If the issue is a metadata inconsistency, additional writes may make recovery harder.

Which OpenZFS version are you running, and does the pool import successfully in read-only mode?