NVME SLOG gives slower write performance

[j2mc]

I'm using fio directly on pve to benchmark my ZFS pool (10x Intel sata SSDs raidz2). On 4k randwrite sync=1 I get about 3k iops, when I add my WD SN750 nvme drives mirrored as log I drop to 1.5k iops. On a different server with 4x Intel sata SSDs striped and mirrored, I get ~2k iops, when I add the same nvme drives as log iops started at about 2k then slowly dropped down to <50 iops! I also watched stats using iostat -v 1, those stats are showing the same numbers as fio and that it's using the SLOG for the sync writes.

I should be getting much higher iops from the NVME drive correct? What is limiting this?

 

[wolfgang]

Hi,

The WD SN750 is a consumer NVMe and has not really high 4k sync capabilities.

I should be getting much higher iops from the NVME drive correct?

NVMe is only the protocol and has compared with SATA lower latency and is better in parallel.
But NVMe does not mean automatic is faster than SATA.

What is limiting this?

The capability to write 4k in sync.
A real good device for this is an Intel Optane Memory.
It is very fast in 4k sync write.

 

[j2mc]

Thanks for the response! You are totally right, I formatted one of the sn750's as ext4 and ran the same fio 4k sync benchmark and got 1.2k iops. The benchmarks I had based my thinking on were showing 4k random writes around 35k iops, which I should have known are async(my async benchmark showed 42k iops).

I thought the raidz2 overhead would slow writes down much more than it actually does. I just ran another couple benchmarks simulating a write heavy db and measuring sync times, here are the results:

Drives	average sync time (usec)	99th percentile sync time(usec)
10x Intel DC sata SSDs raidz2	8.5	53
4x Intel DC sata SSDs raid10(zfs)	266	537
WD SN750 nvme (ext4)	1109	11207
8x 10k SAS raidz2 (7 years old)	6485	20841

The reason I started to investigate this was because of a noticeable slowdown on DB writes, I can now see why. This nvme drive is closer to my old spinning rust than the Intel sata SSDs in a raidz2!