dd if=/dev/zero of=/tmp/test2.img bs=512 count=1000 oflag=dsync
What is your ashift? If you sync write 512B blocks to a pool that is setup with a ashift of 12 (4K block size) you get massive write amplification. Never write blocks that are smaller then your smallest block size. Also your consumer SSDs got no powerloss protection so they can't cache sync writes and because of that writes are slower and can't be optimized prior to writing so your write amplification will explode again.
An example...lets say you try to write a 512B block to a 4K (ashift=12) ZFS pool and that Pool tries to write it to a SSD that is actually working with a 16K block sizes (but is lying and reporting itself as using 512B or 4K LBA) and can only erase blocks ob 128K:
So you write a 512B block. That is smaller than the blocksize ZFS is using so it can'T just write it, it will need to write 4K. Then your SSD will receive the 4K block, cant write it because it works interally with 16K so it will write it as a 16K block. But it can't write a 16K block without erasing a complete 128K block first. So it needs to read seven 16K blocks, erase the full 128K and write the seven 16K blocks + your new 16K again. Because it is done as a sync write the SSDs can't do anything else until this write is complete (so no parallelization possible) and will write one block after another.
So if you write 1000x 512B actually 1000x 128K are read and written. Thats bad for the performance and bad for the wear of the SSD.
With an enterprise SSD that wouldn't be such a big problem because it can still cache sync writes because no data can be lost on an power outage (because of the powerloss protection).
A enterprise SSD would just merge 1000x 4KB as 32x 128K in cache so it needs only to write 32x 128K.
So for the same amount of data (1000x 512B = 512K) a consumer SSD writes for example 128M (1000x 128K) and a enterprise SSD only 4M (32x 128K). Thats why enterprise SSDs are way faster if you got workloads with alot of sync writes.
With the "
dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync" command it isn't that bad because you are just writing a big block where the SSDs can handle it much better because you are not forcing it to write blocks that are smaller than it can handle.
And a raidz1 is also bad for IOPS if you are using these 3 SDDs that way. Here a mirror (raid1) or striped mirror (raid10) is generally recommended as a VM storage because you don't need to do all the parity calculations so there is less overhead. With any raidz1/2/3 you would also need to increase your volblocksize or you are wasting alot of capacity due to padding overhead. And a bigger volblocksize may be bad again for stuff like DBs that do alot of small writes.
By the way...89.4 MB/s for a sync write isn't slow.
And to quote the FAQ of the
official Proxmox ZFS benchmark again:
Can I use consumer or pro-sumer SSDs, as these are much cheaper than enterprise-class SSD?
No. Never. These SSDs wont provide the required performance, reliability or endurance. See the fio results from before and/or run your own fio tests.
So if you can still return the SSDs you might want to get something that is enterprise grade instead if you don't like the sync write performance.
But there it is also important not to get a enterprise SSD that is only rated for read-intense workloads. You want a SSD that is made for write-heavy workloads or atleast mixed workloads.
