SSD slow read and write

[osalj]

Hi,

I built a server for Proxmox with AMD Ryzen 5900X, 64GB RAM ECC DDR4, Gigabyte MC12-LE0 Rev. 1.0, 2x 250GB SSD Crucial MX500 for Proxmox and 2x 1TB Samsung Pro 860 for storage

250GB drives for Proxmox are set to Mirror ZFS, 1TB drives for data storage are also set to Mirror ZFS.

I am very concerned about the writing and reading speed of all drives.
For testing, I used the script https://github.com/TheRemote/PiBenchmarks

Result for disks in ZFS mirror 250GB:


Result for disks in ZFS 1TB mirror:



Before installing Proxmox, I installed Windows 10 on this server.
The 1TB Samsung Pro 860 drives were set to software RAID1 in Windows
Below is the reading and writing speed result from Windows.


What could be the reason that drives have such poor reading and writing performance in Proxmox?

Thanks for any help!

 

[leesteken]

You are using different tests on Windows (I assume CrystalDiskMark?) and Proxmox. ZFS is slow because of its features and is different from software RAID1 (because of the same features). Your drives have poor sync write performance (which you don't test on Windows, I think). Your drive types and ZFS performance has been discussed before several times.
Maybe just use LVM/ext4 or Windows Hyper-V instead if you care about CrystalDIskMark more than the ZFS features?

 

[cheiss]

Hi,

2x 250GB SSD Crucial MX500 for Proxmox and 2x 1TB Samsung Pro 860 for storage

These are both pretty mediorce consumer SSD and are simply not suited for hypervisor usage, especially with ZFS. ZFS needs lots of IOPS and fast fsyncs.
Also, these will go bad rather quickly, as hypervisors (and Proxmox VE) write a lot of data to these disks, constantly.

CrystalDiskMark is not a good benchmark, and comparing ZFS to NTFS (probably) does not say much - ZFS is a whole different beast and both filesystems work very differently. In that case, Windows and the cache layer of the disks probably hide a lot of the bad performance.

You can search in the forum, you will find a lot of posts about this topic.

 

[osalj]

At the moment I only have these SSD drives.

So I understand that I need to use diffrent SSD drives.
Which models will be better for Proxmox and data storage?
Can you provide some example models?

 

[leesteken]

Can you provide some example models?

Search for enterprise SSD

 

[osalj]

A friend has two 960GB Samsung PM893 SSD drives for sale. The drives are new, packed by Samsung. The price per piece is £100

Will test results actually be higher with these drives?

 

[rit1001]

If you are building a home lab environment you can take a look at the basic Samsung QVO drives. Any SSD is a choice based on size, price, TBW (total bytes written) and warranty. TBW and warranty is often combined to create a DWPD (Drive Writes Per Day) figure.

The Samsung drives have a feature called TurboWrite where part of the drive is configured for higher performance and lower data density. This improves the support for the bursty I/O caused by VM deployment and all the file system IO.

Such drives are no panacea, but if you are currently working with 250GB MX drives you will find 1TB QVO are a great improvement at low cost. In terms of their useable life, their write endurance is 300GB per day (0.3 DWPD) over their 3-year warranty. As you seem to be using the MX drives for the OS deployment, rather than VM storage it should be no problem. In comparison, the 1TB 860 pro drives you seem to be using for VM storage have a rating of 600GB per day (0.6 DWPD) over their 5-year warranty.

 

[rit1001]

A friend has two 960GB Samsung PM893 SSD drives for sale. The drives are new, packed by Samsung. The price per piece is £100

Will test results actually be higher with these drives?

Yes and I/O speeds will be far more stable. You will need to check just how much data has been written to these drives. The standard warranty for such drives is 5 years and the DWPD is 1.0 - so 960GB per day.

You can get all the details out of a drive by connecting it to a Windows PC with Samsung's Magician software installed.

 

[justinclift]

This is the data sheet for those drives. The Total Bytes Written (TBW) spec for the 960GB model is 1,752TB. So yep, that's pretty much a full drive write every day for 5 years.

 

[justinclift]

The Samsung drives have a feature called TurboWrite where part of the drive is configured for higher performance and lower data density.

That sounds like the pseudo-SLC thing in consumer level drives, which gives decent performance while the drive is largely not being written to much but turns to extreme crap when the SLC cache runs out. Like when copying several GBs of data around.

 

[justinclift]

@osalj Does the computer you're doing this on have enough room for a PCIe card?

Am kind of thinking (for a homelab situation) that you might want to add a SAS PCIe card and grab some cheap SAS SSDs from Ebay.

Those two Samsung PM893's sound pretty good too though.

 

[Kingneutron]

If you are building a home lab environment you can take a look at the basic Samsung QVO drives. Any SSD is a choice based on size, price, TBW (total bytes written) and warranty. TBW and warranty is often combined to create a DWPD (Drive Writes Per Day) figure.

The Samsung drives have a feature called TurboWrite where part of the drive is configured for higher performance and lower data density. This improves the support for the bursty I/O caused by VM deployment and all the file system IO.

He's already having issues with SSDs and you're recommending QVO?

No, man. Just no.

If you're going to have to re-do it, at least do it right. Proxmox docs recommend Enterprise SSD - for homelab you can get away with less as long as you're aware of the tradeoffs (and probably bottlenecked I/O) but I would avoid QVO drives just as much as SMR.

 

[justinclift]

Doing some looking through Ebay UK, £100 seems to be the cheapest price available for the 960GB model (used). Scan seem to be selling them at that price used too.

But for not much more it's possible to get double that storage capacity with 1.92TB PM893's: https://www.ebay.co.uk/itm/266845619028 : roughly £128, with ~£20 more for postage

 

[justinclift]

Meanwhile, on the SAS path there's stuff like this:

• https://www.ebay.co.uk/itm/335417156068 : 1.92TB for £99.99
• https://www.ebay.co.uk/itm/235506056740 : 1.92TB for £100.00
• https://www.ebay.co.uk/itm/135063587996 : 3.84TB for £199.99
• https://www.ebay.co.uk/itm/256533692981 : 3.84TB for £199.99

(there's a bunch more)

---

And this SATA SSD might be worth trying to snipe just as the auction ends: https://www.ebay.co.uk/itm/126512913819

 

[leesteken]

If you are building a home lab environment you can take a look at the basic Samsung QVO drives.

Please search the forum for QLC drives and ZFS. People get really upset when they find out the money was wasted. And then they ask about good drives and you'll also find recommendations about (second-hand) enterprise drives with PLP.

 

[justinclift]

You will need to check just how much data has been written to these drives.

It was mentioned the drives are new. I guess "how new?" might be the question.

 

[rit1001]

Please search the forum for QLC drives and ZFS. People get really upset when they find out the money was wasted. And then they ask about good drives and you'll also find recommendations about (second-hand) enterprise drives with PLP.

The original post states that the current Crucial MX500 drives are for Proxmox with 1TB Samsung Pro 860 drives handling the storage. Proxmox installs to LVM and has a light load on its disks unless you decide to try and oversize SWAP so that it has to page memory like mad.

 

[leesteken]

The original post states that the current Crucial MX500 drives are for Proxmox with 1TB Samsung Pro 860 drives handling the storage. Proxmox installs to LVM and has a light load on its disks unless you decide to try and oversize SWAP so that it has to page memory like mad.

Mostly yes, but you suggested QVO drives, which are really bad for ZFS because of the QLC.

 

[rit1001]

It was mentioned the drives are new. I guess "how new?" might be the question.

Any open box SSD needs to be checked, at least with Samsung it can be done with an easy to use tool.

 

[rit1001]

Two key things regardless of what type of drives you have, could have or should have.

If you are using ZFS what have you done regarding autotrim and have you considered disabling atime tracking?