In our testing with
ZFS 2.3, Linstor on
NVMe devices, we've achieved impressive results, particularly when leveraging
NVMe-oF (RDMA) over a
100 Gbit ring topology in a three-node cluster, 2x mirror on two
nodes 1 and 2, accessed from diskless
node 3. Our primary storage configurations include
NVMe and
SAS SSDs, with the most significant performance gains coming from
Gen4 and Gen5 NVMe drives - a result that was expected but still remarkable, up to 2x and reliable constant 100GBit wire speed, even more locally.
Key Takeaways from Our Testing
- Diskless Node Access: The real game-changer was accessing storage from a diskless node via NVMe-oF (RDMA), maximizing throughput and minimizing latency.
- Network Efficiency: Running on a 100 Gbit ring topology ensured ultra-low-latency data access across the cluster and is very affordeable.
- Stability & Reliability: Since the beginning of our tests with ZFS 2.3-rc1, we have encountered zero issues - a testament to its robustness.
- Dataset-Level Control: One of ZFS strengths is the ability to control storage behavior dynamically at the dataset level during runtime.
Ready for Production?
Based on our experience, we confidently recommend using
ZFS 2.3 in production environments. The combination of
high-speed networking, NVMe-oF, and Gen5 NVMe drives delivers exceptional performance while maintaining stability. However, this is our personal experience, not a paid endorsement - just a genuine and personal recommendation from the field.
