To make ZFS run well, the top things:
Get enterprise-class drives, SSDs if possible
Use striped mirrors (ZFS equivalent of RAID10), the more disks the better
Make sure ashift is correct
Read and learn how ZFS works. It's complex.
Second-tier things, won't make as much a difference but will...
Pardon me, as I am sure this is common knowledge for many PVE users. For me, it took many frustrating hours to find the fix, so I will document it here as I have not seen this particular problem/solution laid out explicitly elsewhere. This video from the PVE team documents the fix but...