High IO delay and bad zfs raid 1 ssd performance

[tschmidt82]

Hello together,

I have a strange behavior with with a

HP MicroServer Gen10 Plus (CPU E-2246G, 64GB RAM, 2x MICRON Enterprise SSD 7,4TB)
ZFS Raid1 for boot + vms

guest os: 1x windows 2008 r2

at that point where the vm causes i/o's write or read,
the pve is showing high IO delays.

I played around with
compression=off, lz4
atime=off
zfs_arc_max=16gb
zfs_arc_min=8gb

but non of this parameters does really changed a thing

many thx for all input

best regards
TSchmidt

 

[aaron]

What SSD models exactly? Are they, by any chance, QLC ones?

 

[tschmidt82]

Hi Aron, this are indeed 2x Micron 5210 (MTFDDAK7T6QDE) 7.68TB based on QLC Flash Nands (tech specs - link)

do you think this can be the root cause?

 

[Dunuin]

That datasheat is interesting...
0.8 DWPD for 128K sequential writes
0.05 DWPD for 4K random writes

Sounds like the real internal blocksize is quite high if you get a internal write amplification of factor 16 when switching from 128K to 4K writes. You could try to create a pool with ashift=16 (that you be a blocksize of 64K) and look if that make the pool faster.

 

[chrcoluk]

There can be all sorts of reason for i/o delay, a good optimised spindle pool could out perform a badly configured ssd pool. Its not always as simple as throwing hardware($$) at the problem.

Are you able to provide the following information?

Configured ashift of the pool.
primarycache configuration.
Using zvol or dataset with file based storage.
Block size/ Record size
Cluster size on guest OS
Filesystem on guest OS
Qemu cache setting.

Dunuin its for sure higher than 8k given that 16k randoms give more endurance, so yeah 16k+ is my guess at the block size for 8tb model, I think the 4tb might be 8k, and 2tb 4k.

So for the 4tb QLC model, I would go with ashift of 13, also consider 14.

 

[aaron]

do you think this can be the root cause?

Yep. If you look at some benchmarks, QLC SSDs are really bad. As soon as the SLC cache is full, the performance becomes abysmal.

The market for SSDs is a minefield. Especially if you shop in the lower price ranges. Some vendors will change out all the components that make up the SSD and keep the name... See this article for example (German). Or here in English the paragraph about the Pro and Cons, mentioning the same problem.

I consider Micron to be one of the good ones in this regard, but QLC is still not great unless you only want to primarily store data and don't care much about continuous write speeds.