Proxmox with Second Hand Enterprise SSDs

S

silverstone

Well-Known Member
Apr 28, 2018
98
9
48
35
I had many problems related to VERY HIGH iowait on my Proxmox VE Systems using Crucial MX500, which is a Consumer SSD (and one with very little DRAM, so once that fills up, Performance drops like a Rock).

Now I got 2 x Intel DC S3610 1.6 TB SSDs which should be very good for VM Storage or also more intensive stuff, since I got Reccomendation from many Sources that - even second Hand - Enterprise SSDs have WAY better Performance.

According to the Datasheet, these should be rated at 10.7 PB (10'700 TB) Write Endurance.

Looking at the SMART Data, Assuming it wasn't tampered with, shows that there were approx. 4.2 TB of Data Written, although by doing a Pass with badblocks it seems (by analyzing the Differences before/after) that that Calculation is 8x an UNDERESTIMATE, probably due to the wrong Block/Sector Size Used in my Calculation (probably a 512b vs 4096b Issue):
Code: 
echo "GB Written: $(echo "scale=3; $(sudo /usr/sbin/smartctl -A /dev/disk/by-id/ata-XXXXXXXXXXXXXXXXXXXXXXX | grep "Total_LBAs_Written" | awk '{print $10}') * 512 / 1073741824" | bc | sed ':a;s/\B[0-9]\{3\}\>/,&/;ta')"

Anyways, the main Concern is the HUGE RAW Value for Raw_Read_Error_Rate and Read_Soft_Error_Rate.

On one Disk Raw_Read_Error_Rate went from 3211595278 to 4294967295.

Badblocks didn't report any Bad Blocks but it still doesn't inspire Confidence. It might be Flash/Controller Calendar Aging, because surely it's not Wearout at 0.3% of the Endurance Rating :( .

Not sure if I should Return these to the Seller (it's going to be quite Expensive to ship these back to the U.S.) or what to do with them. Should i try to put them in a ZFS Pool, do some fio on it and kind of stress Test ?
 

Attachments

beisser said:
i have the same ssds and my numbers there are huge as well. i personally wouldnt worry about those.
the link to backblaze is good for hdds, not so much for ssds.
Click to expand...
So you can confirm VERY high values for the same Attributes (Raw_Read_Error_Rate and Read_Soft_Error_Rate) ?

What about the Reallocated Sector Count which I have at 1 or 2, should I be worried about that ?

If I could trust the SMART Attribute 180 the Unused_Rsvd_Blk_Cnt_Tot is set at around 35887 so there should be still some to Spare.

SMART Attribute 179 Used_Rsvd_Blk_Cnt_Tot is set to 1/2 which matches Reallocated_Sector_Ct.

Maybe I could do a bit of Overprovisioning ? I normally don't do "real" Overprovisioning by resizing via hdparm, but I just create a smaller Partition than Full-Size Drive. I'm not sure if this really counts as Overprovisioning though (well, the OS / Linux Kernel will still "think" that the available space is lower than Rated, it's just NOT hidden the way Overprovisioning via hdparm usually works).
 
I have a few customers who are using only used SSDs in CEPH cluster. Just buy a few spares, and when they get kicked out of CEPH cluster just replace them.
 
  • Like
Reactions: silverstone
the reallocated sector count is something to be more worried about, but the number 1 or 2 by itself has little meaning, other than stating that 1 or 2 sectors were reallocated at some point in time.
important (in my opinion) is the rate at which it increases (or not).
lets say you have 2 reallocated sectors and a year later it increases to 3.
that would be absolutely nothing to worry about.
if it increases by 1 every other day it is indeed something to worry about as the ssd only has a limited amount of sectors to reallocate (in case of the s3610 about 36000 unless i completely misinterpret smart data).

Code: 
179 Used_Rsvd_Blk_Cnt_Tot   0x0033   100   100   010    Pre-fail  Always       -       0
180 Unused_Rsvd_Blk_Cnt_Tot 0x0032   100   100   000    Old_age   Always       -       35891

this is the values for reserved blocks used and available for my drive.
my guess is that yours shows 1 or 2 blocks used and the unused number 1 or 2 less than mine.

unfortunately both my drives are perfectly healthy so i dont have any practical numbers to compare.
but i wouldnt worry about the reallocated sectors on the ssd if they are as low as 1-2 and dont increase in short intervals.

edit: here are my large numbers so that you see that they are pretty similar

Code: 
 1 Raw_Read_Error_Rate     0x000e   130   130   039    Old_age   Always       -       3752936212
  5 Reallocated_Sector_Ct   0x0033   100   100   001    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       11209
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       59
 13 Read_Soft_Error_Rate    0x001e   130   130   000    Old_age   Always       -       3752936212
 
Last edited:
  • Like
Reactions: silverstone
silverstone said:
Not sure if I should Return these to the Seller
Click to expand...
You bought used drives, and the numbers smart reports dont mean what you think they mean. 4.2TBW on write optimized drives (which yours are) means these are basically brand new.

What do you intend to actually do on these? If its just typical home lab stuff they will do just fine. and if/when they fail, replace them.
 
alexskysilk said:
You bought used drives, and the numbers smart reports dont mean what you think they mean. 4.2TBW on write optimized drives (which yours are) means these are basically brand new.
Click to expand...
Well, I think they are marketed as Mixed Use (whereas the S3700/S3710 was for more Write-Intensive Workloads).

But yes, agreed, 4.2TB (or "4TB" for short just to avoid Confusion with the Decimal / 1k Separator <.> vs <,> vs <'>) - which might actually be 34TB due to the wrong Sector/Block size (512b vs 4096b used in the Calculation, since writing 1.6TB pass with badblocks increased it only by 200GB or so), but even at 34TB, that is Peanuts - absolutely nothing for such a Drive rated for 10PB (or 10'000 TB or 10k TB).

It's like Brand New except for the Ton of Read Errors ...

I had a quick 5 seconds Test or so with f3probe (that's how fast it was) and it passed as Genuine. Maybe worth trying with f3write and do a full Write Pass to Validate the Capacity for Real to make sure it's not Fake ?

alexskysilk said:
What do you intend to actually do on these? If its just typical home lab stuff they will do just fine. and if/when they fail, replace them.
Click to expand...
My Target was for VM Storage in Proxmox (what I usually do), where I was really struggling with the Crucial MX500 SSD due to high iowait Time of >85% (my Assumption is that after filling up the DRAM Cache of 512MB [!!!] the Drive Performance drops like a Stone).

For Reference (which is probably a bit skewed due to the Write Amplification I was getting with a Guest VM with ZFS on top of the Host Virtual Disk ZVOL) I had approx. 160 TBW in around a Year on that System (LUKS + ZFS Host + ZFS on ZVOL in VM), although on Similar Systems it can be quite lower, thus indicating that it might be a ZFS on ZVOL HUGE Write Amplification Issue.

On another System I had approx 28TB in 0.5 Years so say 56TB/year.

I cannot exclude that these Workloads will not increase in the Future. But it should still be Peanuts for such a Drive :) .

If they are "too Risky" for such a workload, I could maybe use them as L2ARC instead (I never used L2ARC but it's maybe one scenario where Failure is NOT too Critical).
 
Don't worry too much with the math or prediction. The choice of enterprise SSDs for high write loads (Ceph) and mid-range ones for low write loads (boot disks, generic storage) is the main thing.

My conclusion over the years with HDDs and SSDs of all brands/sizes/types/firmware:
buying used parts is perfectly fine
no matter how expensive or cheap, they will die at some day, some sooner, some later

Just make sure you have enough replacements on the shelf (tip: buy countercyclically and always look out for super cheap offers). Regardless of whether you use ZFS, Ceph or hardware RAID, it has always been good advice to dimension it so that 2+ disks can fail without any problems and to use different models/manufacturers of a similar class for each vdev or RAID group. This generally minimizes the risk of headaches and if a disk does break, you are prepared and can simply replace it like a boss (without panicking and then ordering replacements, possibly at an inflated price).
 
  • Like
Reactions: silverstone and Johannes S
they are not too risky.

some of the smart values are simply irrelevant or proprietary encoded.

see https://serverfault.com/questions/854770/raw-read-error-rate-for-intel-ssds

dont look at the raw value, look at the values before:

Code: 
  1 Raw_Read_Error_Rate     0x000e   130   130   039    Old_age   Always       -       3794021404
  13 Read_Soft_Error_Rate    0x001e   130   130   000    Old_age   Always       -       3794021404

as you can see here i have insanely high raw numbers in the end, but the "value, worst and threshhold" numbers indicate everything is completely ok.
 
  • Like
Reactions: silverstone and Johannes S
mr44er said:
Don't worry too much with the math or prediction. The choice of enterprise SSDs for high write loads (Ceph) and mid-range ones for low write loads (boot disks, generic storage) is the main thing.

My conclusion over the years with HDDs and SSDs of all brands/sizes/types/firmware:
buying used parts is perfectly fine
no matter how expensive or cheap, they will die at some day, some sooner, some later

Just make sure you have enough replacements on the shelf (tip: buy countercyclically and always look out for super cheap offers). Regardless of whether you use ZFS, Ceph or hardware RAID, it has always been good advice to dimension it so that 2+ disks can fail without any problems and to use different models/manufacturers of a similar class for each vdev or RAID group. This generally minimizes the risk of headaches and if a disk does break, you are prepared and can simply replace it like a boss (without panicking and then ordering replacements, possibly at an inflated price).
Click to expand...
But what SSD would you reccomend besides the Intel S3610 ? I have lots of space for SATA not so much for PCIe (or M.2, let alone U.2) since I don't really have Slots / Caddies for those (in the Supermicro Chassis) or PCIe Lanes available (in Desktop-Like chassis such as a Fractal Define R4/R5/XL R2/etc).

I was eyeing the Micron 5300 Pro/Max (maybe also the 5200 ?). Samsung not so sure about their quality. Toshiba not so sure.
 
my opinion: any enterprise-ssd is acceptable as long as it fits your needs (capacity, endurance and most importantly price).
no need to pay for something insanely expensive from vendor a if a comparable ssd from vendor b is available cheaper.
the only thing you should do is keep their firmwares up to date.
a good resource of knowledge for second hand enterprise ssds (and hdds) is servethehome where there are lots of dicussions around the topic and even firmwares get published: https://forums.servethehome.com/index.php?forums/hard-drives-and-solid-state-drives.17/

i often read the forums there to keep my knowledge up to date on those things.
can definitely recommend.
 
  • Like
Reactions: Johannes S
beisser said:
my opinion: any enterprise-ssd is acceptable as long as it fits your needs (capacity, endurance and most importantly price).
no need to pay for something insanely expensive from vendor a if a comparable ssd from vendor b is available cheaper.
the only thing you should do is keep their firmwares up to date.
a good resource of knowledge for second hand enterprise ssds (and hdds) is servethehome where there are lots of dicussions around the topic and even firmwares get published: https://forums.servethehome.com/index.php?forums/hard-drives-and-solid-state-drives.17/

i often read the forums there to keep my knowledge up to date on those things.
can definitely recommend.
Click to expand...
Would you be able to give a quick Summary? To read that Forum Section seems like a Full Time Job :eek:.

There were some Auctions of NVME PCIe x8 Samsung MZ-PLK3T20 3.2TB as well as NVME PCIe x8 Sandisk SDFADAMOS-3T20-SF1 ioMemory 3.2TB Toda,y but (even though they were sold quite Cheap, 125 EUR/piece for the former, 65 EUR/piece for the latter) I wasn't quite sure about how good they really were.

I'm mostly looking for SATA SSDs, but for some Systems, if/where I have PCIe Lanes and Slots to spare, I won't say no to NVME :) .
 
not sure what you expect here. but asking to write a summary is like asking to give summary of the proxmox forums (or any of its sections).
if you are interested in getting a specific model, see if that model gets mentioned there and if they have any firmware-bugs that require an update (some do, some dont).
usually if there are updates necessary someone provided the binary file for it in the respective threads.
if not, its not hard to ask for it.

also you dont need to read everything. im not interested in the hdd stuff for example so i skip those threads and only read interesting looking ssd threads.
 
For ssd/nvme you need option "-x" instead of "-a" like eg "smartctl -x /dev/nvme0n1" to see the used life endurance
which is the most important health indicator for kind of ssd disks !!
 
Last edited:
  • Like
Reactions: Johannes S
silverstone said:
But what SSD would you reccomend besides the Intel S3610 ? I have lots of space for SATA not so much for PCIe (or M.2, let alone U.2) since I don't really have Slots / Caddies for those (in the Supermicro Chassis) or PCIe Lanes available (in Desktop-Like chassis such as a Fractal Define R4/R5/XL R2/etc).

I was eyeing the Micron 5300 Pro/Max (maybe also the 5200 ?). Samsung not so sure about their quality. Toshiba not so sure.
Click to expand...
Please do not take this as a recommendation, because I do not have enough long-term experience with NVMes or even many of them in use to give good purchasing advice.;)

I personally use Micron7400 and 7450 because they were mentioned often in the German-speaking forum, especially in relation to Ceph. Accordingly, they were difficult to buy and when a good offer came around, I jumped at the chance.
No problems so far, running and performing great.

silverstone said:
Samsung not so sure about their quality. Toshiba not so sure.
Click to expand...
As said, be prepared that disks can die at day1. As I said, make sure you have enough redundancy yourself (also think about the postal service and how long it will take in the event of a warranty claim if you need to replace an NVMe) and never rely solely on any quality promise from a manufacturer. No matter how great their warranty service is, once the data is gone, it's gone.
 
  • Like
Reactions: silverstone
mr44er said:
Please do not take this as a recommendation, because I do not have enough long-term experience with NVMes or even many of them in use to give good purchasing advice.;)

I personally use Micron7400 and 7450 because they were mentioned often in the German-speaking forum, especially in relation to Ceph. Accordingly, they were difficult to buy and when a good offer came around, I jumped at the chance.
No problems so far, running and performing great.


As said, be prepared that disks can die at day1. As I said, make sure you have enough redundancy yourself (also think about the postal service and how long it will take in the event of a warranty claim if you need to replace an NVMe) and never rely solely on any quality promise from a manufacturer. No matter how great their warranty service is, once the data is gone, it's gone.
Click to expand...
And about SATA/SAS, anything good there in your Opinion ? I really don't have that many slots for NVME (if any).
 
For proxmox i would use : Samsung, Micron, Intel and probably Toshiba/Kioxia , of enterprise flavour. This is it. Of course, have backups,have redundancy and this is it.
 
  • Like
Reactions: silverstone
I have very good experience with HGST spinning rust (consumer and enterprise), but WD bought them. I don't know if that had any impact on quality at all and how their SSDs behave.

Another good shot should be Seagate Nytro or anything from KIOXIA.

For not that important cases or less IOPS demanding stuff I use cheap Intenso SSDs and triple redundancy...
 
You must log in or register to reply here.
Top Bottom