New host with SATA

[Alessandro 123]

In addition to the other server that I'm planning, with SSD only, I need a second server with SATA disks and SSD as zil/l2arc

I'll go with a 3way mirror, which are the most reliable SATA disk based on your experience?

Are wd gold good enough?

 

[LnxBil]

As long as you use enterprise class sata disks (or ones with enterprise firmware), you're good to go. I cannot recommend disks, because SATA fail a lot more than the usual SAS counterparts (not NL-SAS) and I mainly use those.

 

[Alessandro 123]

I also use SAS but SAS are much more expensive and much more hotter than SATA and also their smart details are much more strange

With SATA you have a standardized output with multiple parameters to look for, in SAS you don't have this and probably the only parameter to read is the grown list

Also SAS disk are much more difficult to find, in a rush, an enterprise SATA disk is sold everywhere, SAS usually must be ordered

I think that SAS disk are slowly fading out due the lower price of SSD and their greatest performance

 

[LnxBil]

SAS and SATA are protocols, SSD is a flash technology, so you cannot compare. Currently, there are at least 3 types (I know of) of solid state drives available: SATA (including mSATA), SAS and PCIe-based (like NVMe) ordered by performance and also price.

Normally, I do not build servers "in a rush", so I don't care if SAS is not available at the store around the corner, it is the better and faster technology over SATA. I also do not care about disk failures, therefore on has warranty - as long as the disks do not fail all together :-D

 

[Alessandro 123]

I know that are different protocol, but I was semplifing.

I don't build server in a rush, but when all is going bad, you have to replace multiple disks and you must have multiple disks available in house or your vendor warehouses. Last year I've totally lost 2 servers due to multiple disks failure during a rebuild (both RAID10, stupid raid level) and a third triggered a second failure during another rebuild (hopefully was a RAID6).

2 weeks ago the same happened: disk failed, rebuild, 97% another disk was failed.

All of these server was with enterprise SAS disks. Usually I don't use SATA on servers, but I don't think SAS is the way to go, in 2017.
SSDs are a little bit more expensive than SAS 15k (in our case, a 480GB Si3610 cost about €300,00 and a 300GB SAS 15K about €270,00)
but much faster and doesn't require any special hardware (SAS require a SAS controller) and are much colder (our 600GB SAS are near 40-43°C, near 48-49° during a rebuild)

Anyway, probably I'll use Gluster, so all data would be replicated on 3 different hosts and probably with a RAID-1 on each server (in other words, the same data is replicated 6 times.). 6x4TB SATA disks is still MUCH cheper than the same raw capacity with SAS

Yes, you care about failures because disks die in batches...........


OT: do you use Seagate SAS ? If yes, I have a question for you.

 

[fireon]

I can say for big data we have good experience with the SATA WD Red Pro.

2 weeks ago the same happened: disk failed, rebuild, 97% another disk was failed.

If you have Smartcheck enabled... for example we do this with check_mk, then this should never happen. On Smart you have all informations that you can change your disk a little pit before they crash. CRC Errors and others. ZFS send out a long time before smart detect some errors.

 

[JTY]

I've had great luck with Seagate ES drives.

 

[Alessandro 123]

As long as you use enterprise class sata disks (or ones with enterprise firmware), you're good to go. I cannot recommend disks, because SATA fail a lot more than the usual SAS counterparts (not NL-SAS) and I mainly use those.

Do you think that this: ST2000NM0045 is to be considered reliable as SAS or SATA? Capacity, speed, MTBF and URE are the same as enterprise SATA, but interface is SAS

 

[Alessandro 123]

If you have Smartcheck enabled... for example we do this with check_mk, then this should never happen. On Smart you have all informations that you can change your disk a little pit before they crash.

I have smart monitoring on all server, but SMART is not always reliable.

ZFS send out a long time before smart detect some errors.

What do you mean with "ZFS send out a long time before smart detect some errors" ?
ZFS fails a disk long time before SMART detect issues ?

 

[LnxBil]

Do you think that this: ST2000NM0045 is to be considered reliable as SAS or SATA? Capacity, speed, MTBF and URE are the same as enterprise SATA, but interface is SAS

I cannot tell. I don't use NL-SAS drives, only "real" SAS drives - so for disks 10k up and SSD. Also, I use only drives in servers with warranty and a service plan so if something breaks, I'll get it fixed at most NBD (next business day).

 

[Alessandro 123]

Warranty doesn't increase reliability, it only helps lowering the TCO. If you have to buy a new drive on each failure, that kind of driver/vendor has a TCO higher than a competitor that replace for free.

Anyway, SAS 10/15k has very low capacity compared to SATA.

 

[LnxBil]

Warranty doesn't increase reliability, it only helps lowering the TCO. If you have to buy a new drive on each failure, that kind of driver/vendor has a TCO higher than a competitor that replace for free.

Yes, of course. I just wanted to say that I don't care what drives are in my system as long as my vendor thinks they're good :-D

Anyway, SAS 10/15k has very low capacity compared to SATA.

Yes, but I'm in the database business (OLTP) and there is nothing more important than throughput and latency, so go for the faster drives.
And the difference between SAS 10k and SAS 7.2 in the 2.5'' segment is not that big, for 3.5 yes, but I have not used 3.5 disks in servers for over half a decade. You can nowadays stuff 24x 2.5'' disks in the front of a 2 HE server and often also some in the back, so that's totally enough for performance and capacity. We also reached 2.5'' SSDs with capacity over 2 TB. Even the newly built NVMe 2.5'' disk drives fit in nicely. Currently, there are not enough PCIe lanes available for powering everything up without using some kind of multiplexers, but hey, the new AMD server processor has 128 PCIe lanes :-D

In the old "real" harddisk days with mechanical spindels, you often have too much storage because you needed a lot of IOPS. Nowadays that shifted a lot by using SSDs. We use the "big" 1.2 TB SAS 10k 2.5 drives for backup and there is also the throughput more important than storage capacity.

 

[Alessandro 123]

Can you share some vendor/models ?
I would like to check prices with our distributor

 

[LnxBil]

The ones I can access right know are Seagate ST1200MM0088. We use mostly Fujitsu and HP hardware and the the others like Cisco, Dell (and rebranded Bull), IBM, Supermicro.

 

[Alessandro 123]

Can you tell me when, usually, you replace a SAS disks? Do you wait for "predicted failure" triggered by SMART or when value of Elements in grown defect list starts to increase ?

 

[LnxBil]

If the failure gets predicted, we swap them normally immediately, yet most of the time, the disks fail without any prediction. We often have at least one spare on storage, if we do not have a hot spare, so that we do not have to wait for the next day to get a replacement.

 

[Alessandro 123]

Like us but based on which parameters do you "predict" a failure?

Only smart "predicted failure" attribute or even reallocations? How many reallocations makes you swap the disk?

 

[LnxBil]

On the not-ZFS-based sets, we usually use the raid controller and it predicts the failure and warns us. Same on our SANs.

For our ZFS pools, we do not monitor specific values, we use the "default" nagios/icinga smart check for that. Normally, we use simple SATA drives there, so we "see" problems with increased response times of the drives, we know to swap them even before any smart error occured.

 

[Alessandro 123]

Raid controllers marks a disk for replacement only after failure or after smart has triggered the "predicted failure"

So you won't replace after some reallocations