Western Digital Red NAS SATA SSDs - High latency

[mcdowellster]

Hello All,

I've been getting the infamous warning about slow reads and ops on bluestore but only from these two (my NEWEST SSDs).
OSD 7 and OSD 2 are 4TB WD Red NAS SATA SSDs. The rest are a mix of Kingston DC SATA SSDs and Samsung 860 Pros @ 2TB.
This is causing IO delays.


Yes before I get all the hate, it's a 2:1 SSD Pool. Yes... Two services with IDENTICAL OSDs (model, size and number) and are exactly the sameon each server.
Yes there is enough RAM (384GB each)
Yes there is enough CPU (Intel(R) Xeon(R) Silver 4210R CPUx2) - 2.85% usage
These are on a 10Gbit DAC connected SAN on a Unifi Agg 10Gbit switch. It's nowhere near network capacity. There are three servers (extra for Q + processing for services), all connected with Intel NICs, jumbo frames (recent troubleshooting) and full logging to Graylog. Nothing is dropping, no errors, it's clean.

Virtual Environment 8.4.8
CEPH: 18.2.7

Now I realize that they have twice the PGs the other SSD drives do which naturally would mean more read and writes. I'm not seeing activity that would suggest it's anywhere NEAR the maximum of even a single SATA SSD.

Short of throwing these expensive ($400 CAD each) things out, is there any other reason why these would be completely beaten down by Ceph?
I've already adjusted mClock and tried several tuning guides. Just these two disks are causing so much IO delay.



Current ops:


OSD Details:


SMART Data

 

[mcdowellster]

Bumps are allowed, right?

 

[mcdowellster]

Heeelllpppp

Please don't ask me to throw away these drives...

 

[Impact]

Check the table here, the recommendations here and this (ZFS related but still) test but just so you can see the difference between normal SSDs and DC ones

 

[mcdowellster]

Check the table here, the recommendations here and this (ZFS related but still) test but just so you can see the difference between normal SSDs and DC ones
View attachment 90970

Thanks for this. Yeah I know... It's just amazing to me that a WD BLUE is out performing a "NAS" drive. It makes no sense to me.

 

[janus57]

Hi,

Specs wise, the SA500 is more a consumer drive than an enterprise drive (usage of pseudo-cache and ~0.35 DWPD)

Cf : https://www.techpowerup.com/ssd-specs/western-digital-sa500-500-gb.d836

A refurbished enterprise SSD would be better

Best regards,

 

[mcdowellster]

Comparing WD500 vs SEDC500

WD500

/dev/sdl: (groupid=0, jobs=1): err= 0: pid=2725304: Wed Sep 24 12:37:33 2025
  write: IOPS=140, BW=561KiB/s (574kB/s)(164MiB/300348msec); 0 zone resets
    slat (usec): min=2, max=1456, avg=14.09, stdev=12.07
    clat (nsec): min=1226, max=720941k, avg=2957074.71, stdev=32007869.44
     lat (usec): min=44, max=720970, avg=2971.17, stdev=32008.80
    clat percentiles (usec):
     |  1.00th=[    44],  5.00th=[    45], 10.00th=[    46], 20.00th=[    51],
     | 30.00th=[    60], 40.00th=[    61], 50.00th=[    64], 60.00th=[    72],
     | 70.00th=[    85], 80.00th=[   161], 90.00th=[   717], 95.00th=[  1745],
     | 99.00th=[ 36439], 99.50th=[ 40109], 99.90th=[583009], 99.95th=[692061],
     | 99.99th=[708838]
   bw (  KiB/s): min=    8, max= 4536, per=100.00%, avg=592.33, stdev=1269.95, samples=569
   iops        : min=    2, max= 1134, avg=148.07, stdev=317.47, samples=569
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=19.28%
  lat (usec)   : 100=59.68%, 250=1.23%, 500=8.92%, 750=1.96%, 1000=2.33%
  lat (msec)   : 2=2.29%, 4=0.56%, 10=0.06%, 20=0.12%, 50=3.17%
  lat (msec)   : 100=0.08%, 250=0.01%, 500=0.01%, 750=0.28%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=750, max=1593.4k, avg=7102.87, stdev=42852.33
    sync percentiles (usec):
     |  1.00th=[   791],  5.00th=[   824], 10.00th=[   832], 20.00th=[   840],
     | 30.00th=[   848], 40.00th=[   848], 50.00th=[   865], 60.00th=[   914],
     | 70.00th=[   938], 80.00th=[  1647], 90.00th=[  3130], 95.00th=[ 35390],
     | 99.00th=[ 77071], 99.50th=[ 98042], 99.90th=[750781], 99.95th=[767558],
     | 99.99th=[792724]
  cpu          : usr=0.22%, sys=0.56%, ctx=84692, majf=4, minf=15
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,42107,0,42106 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=561KiB/s (574kB/s), 561KiB/s-561KiB/s (574kB/s-574kB/s), io=164MiB (172MB), run=300348-300348msec

Disk stats (read/write):
  sdl: ios=44/110243, merge=0/7911, ticks=2582/1809555, in_queue=2404901, util=99.61%

SEDC500

/dev/sdo: (groupid=0, jobs=1): err= 0: pid=2731937: Wed Sep 24 12:42:57 2025
  write: IOPS=911, BW=3646KiB/s (3733kB/s)(1068MiB/300001msec); 0 zone resets
    slat (usec): min=2, max=1886, avg=14.01, stdev=10.40
    clat (nsec): min=912, max=101842k, avg=77389.69, stdev=225887.52
     lat (usec): min=42, max=101851, avg=91.40, stdev=226.66
    clat percentiles (usec):
     |  1.00th=[   42],  5.00th=[   43], 10.00th=[   44], 20.00th=[   48],
     | 30.00th=[   56], 40.00th=[   59], 50.00th=[   61], 60.00th=[   63],
     | 70.00th=[   74], 80.00th=[   82], 90.00th=[   85], 95.00th=[   94],
     | 99.00th=[  445], 99.50th=[  578], 99.90th=[ 1205], 99.95th=[ 1483],
     | 99.99th=[ 4817]
   bw (  KiB/s): min= 1736, max= 4248, per=100.00%, avg=3650.46, stdev=379.56, samples=599
   iops        : min=  434, max= 1062, avg=912.44, stdev=94.86, samples=599
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=22.56%
  lat (usec)   : 100=73.40%, 250=0.83%, 500=2.59%, 750=0.23%, 1000=0.20%
  lat (msec)   : 2=0.14%, 4=0.02%, 10=0.01%, 20=0.01%, 250=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=715, max=109188, avg=1079.01, stdev=611.06
    sync percentiles (usec):
     |  1.00th=[  766],  5.00th=[  906], 10.00th=[  922], 20.00th=[  938],
     | 30.00th=[  955], 40.00th=[  971], 50.00th=[ 1004], 60.00th=[ 1029],
     | 70.00th=[ 1057], 80.00th=[ 1090], 90.00th=[ 1270], 95.00th=[ 1516],
     | 99.00th=[ 2311], 99.50th=[ 4228], 99.90th=[ 5145], 99.95th=[ 6063],
     | 99.99th=[16319]
  cpu          : usr=1.34%, sys=3.67%, ctx=553190, majf=0, minf=42374
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,273418,0,273417 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=3646KiB/s (3733kB/s), 3646KiB/s-3646KiB/s (3733kB/s-3733kB/s), io=1068MiB (1120MB), run=300001-300001msec

Disk stats (read/write):
  sdo: ios=293/573496, merge=0/5514, ticks=181/323307, in_queue=615417, util=96.33%

The WD500 is absolutely terrible in comparison... It's hard to believe that a NAS drive performs this much worst but the numbers do not lie...

 

[alexskysilk]

Not just terrible. likely defective. did you run a smart long test?

 

[mcdowellster]

I'm RMAing these. I've got four 860 Pros in the box still I'm going to slap in for now instead. What a pain in the rear.