Help diagnosing ZFS storage issues

[magic_ren]

I'm going to go out of the box here - not that I disagree with most of the excellent advice given above - but what stands out to me is, and I quote your original post:

I find it interesting that till a month ago you saw NO problems. I would imagine that the problems/suggestions above should have built up over time - & you should have seen at least some form of gradual degradation. (I very much know that I may be wrong with this assessment - but its my hunch).

However, I would lean to the kernel change to 6.8 (which I imagine you updated to about a month ago) which has proven to have its fair share of problems. You should search these forums on that subject - and maybe try pinning to a previous kernel to see if that fixes your issues.

PLEASE ignore this whole post if either you are not on the above kernel or you know for a fact that I'm wrong with your specific HW/setup etc.

thanks mate, appreciate the thought however I'm still on 6.5, it's possible the issues have been going on for a bit longer but i've only really noticed the lock ups recently, all I have to go on is anecdotal evidence until I started investigating properly. I will add however I do feel like performance was better before I upgraded to v8.

 

[Ramalama]

That's a good advice from gfngfn256.
Worth definitively a try.

But tbh, i think now that a drive is somehow dead or have huge issues with writing/deleting.
Maybe even more as one, but let's see.

At least if you get your data moved somewhere else (backup etc) and destroy the pool, you can then speedtest each drive with the script i provided. Or speedtest however you want, to find the bad one.

I think you can view the latency of every drive with zpool iostat, in theory one should have a really huge write latency. Just write during monitoring something to the pool.

But anyway i would backup the data somehow, the risks getting higher that at least one drive is almost dead.

Cheers