High IODelay after replacing HDD with SSD

[slave2anubis]

Hello good people. I am new to this forum (and Proxmox for that matter), i currently have a home server type of setup, where i try to play with virtualization stuff before i apply for a sys-admin job.
My setup is a AMD Ryzen 1700 paired with 16GB ECC RAM, running the latest Proxmox VE.

I have this server setup for over half a year now running on Proxmox, and before that i was running it on a simple Ubuntu server.
A few weeks ago a friend of mine (who buys and sells sh IT equipment) gave me a pair of 480GB Intel 535 SSD that where used in a production server for a few years, he said they a in good condition (90%+ life) and they where very cheap (35$).
Before this i used two HGST 7k6000 4TB drives in mirror configuration for my main ZFS pool, but since i was not using a lot of the capacity (~100GB), i thought that i can make a good power and noise saving switching the pool to the Intel SSD's i just bought, in a similar two drive mirror configuration.

After i created a new ZFS pool from the two Intel 535 drives in a mirror config and moved all the container and VM images, everything looked and worked ok. But there was something strange about the IO delay graph, all of a sudden the IO delay pattern grew two fold compared to the HGST HDD's.

I am not saying that this is a problem since the server overall has a very small load on it, and i don't think it will impact performance very much.
So after a 2-3 week with the Intel SSD's it kept nagging me what the problem was, since i searched the internet's for a explanation but found none. I complaint to my IT friend about this issue and he said he can replace the SSD's with two Samsung PM863a 480GB models, also used in a server environment.
I took the Samsung SSD's and replaced the Intel SSD's one by one in the ZFS pool, and after that the IO delay went way down, lower even than the HGST HDD period.


I know this is not a "real" problem for you guys, but i would like to know why this is happening, since i am new to this virtualization thing and want to understand the causes better.



Thanks you and hope i did not bother you with my post.

 

[LnxBil]

As far as google can answer what the Intel 535 is (never heard from it), it is a consumer SSD, so that is always a no-go for a fast ZFS system. The PM863 is enterprise grade hardware and fast-as-hell in comparison to any consumer grade SSD I've ever seen. Avoid at all costs using a consumer SSD in ZFS if you want a fast system.

 

[slave2anubis]

Hello, thank you for the answer!
As logical as it seems (enterprise vs consumer), i wanted a bit more of a technical answer, because in my mind there is a clear performance difference between the two drives types, but at the same time i am inclined to believe that a SSD (even a consumer one) should outperform a enterprise grade HDD.

I've read a post over at https://calomel.org/zfs_raid_speed_capacity.html where they are testing ZFS Raid modes and performance using consumer grade SSD's (Samsung 840 series) so the argument that consumer SSD's are so much worse then enterprise ones is a bit off in my opinion.

The IO Delay difference between the Intel SSD and the Samsung is ~0.3 vs ~0.01, that is a 30 times difference!

 

[apoc]

The HGST HDD you were using. Was it SAS or SATA attached?

 

[slave2anubis]

The HGST HDD you were using. Was it SAS or SATA attached?

All were connected via the onboard SATA controller.

 

[Bengt Nolin]

Consumer drives often depend on DRAM caches of a few GB to get good performance which means they can handle smaller bursts of data that can get buffered in the cache and then written to the (relatively) slower flash memory. If you fill the cache the disk has to flush the data to flash synchronously, and these kinds of drives often degrade a lot both latency and throughput wise when that happens since they have simpler and cheaper controllers and less parallel memory and so on.

This is often a problem when people try for example Ceph or ZFS using consumer drives.

Those Samsung 863 disks are enterprise grade disks and pretty good ones too.