RAID 5 poor performance compared to RAID 1

ferranms · Mar 13, 2015

Hello all,
after being reading lots of threads on this forum and testing lots of things and configurations, I cannot get RAID 5 (three SSD disks) with better performance than this (poor fsyncs result):

CPU BOGOMIPS: 44799.44
REGEX/SECOND: 1505667
HD SIZE: 19.56 GB (/dev/mapper/pve-root)
BUFFERED READS: 641.59 MB/sec
AVERAGE SEEK TIME: 0.13 ms
FSYNCS/SECOND: 439.09
DNS EXT: 68.18 ms

Same server, with RAID 1 (two SSD disks) I get much better fsync performance:

CPU BOGOMIPS: 44798.88
REGEX/SECOND: 1444152
HD SIZE: 456.70 GB (/dev/mapper/system-root)
BUFFERED READS: 502.84 MB/sec
AVERAGE SEEK TIME: 0.12 ms
FSYNCS/SECOND: 4764.37
DNS EXT: 15.51 ms

I have dell PERC H310 raid controller (no write back). Tested both in ext3 and ext4 with more or less the same results.

Is something wrong in my RAID 5 configuration or is it normal?

m.ardito · Mar 13, 2015

it's quite normal that raid 5 is much slower than raid 1...
but that value seems too low to me... I have raid5 on a lan nas (spinning disks) and I get higher fsyncs

pveperf /mnt/pve/pve_ts879/
CPU BOGOMIPS: 72531.44
REGEX/SECOND: 945994
HD SIZE: 11081.12 GB (ts879:/PVE)
FSYNCS/SECOND: 1891.62
DNS EXT: 58.52 ms
DNS INT: 0.94 ms

local storage (server is ibm xseries from 2010, same as the two sas local spinning disks in raid 1)
pveperf
CPU BOGOMIPS: 72531.44
REGEX/SECOND: 948926
HD SIZE: 16.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 144.61 MB/sec
AVERAGE SEEK TIME: 4.27 ms
FSYNCS/SECOND: 3314.82
DNS EXT: 59.08 ms
DNS INT: 0.91 ms

it could depend on other factors.

Marco

kbrault · Mar 13, 2015

I think it may be hardware related.

I have not heard good things about the H310 in RAID 5 mode.

We have old HP P400's and the RAID 5 is faster than RAID 1 (I do not have the numbers handy) but RAID 6 falls of the proverbial performance cliff.

LnxBil · Mar 15, 2015

One first thing: Please consider NOT using RAID 5 with SSD. The write throughput of RAID 5 is still very heavy on SSD and at least my hardware distributors recommend only RAID 10 with SSDs.

RAID 5 is still the fastest for read performance if and only if the hardware raid controller cannot read from all disks in RAID 10, yet write performance is worse than RAID 10. Please consider also trying a more sophisticated benchmark for the disk performance, e.g. the flexible IO benchmark (fio).

Comparing the two different times you provided, is it possible that the RAID 5 was still in rebuild? What about the Write Back cache and its BBU settings?

ferranms · Mar 15, 2015

Thank you all for your replies.

LnxBil said:
One first thing: Please consider NOT using RAID 5 with SSD. The write throughput of RAID 5 is still very heavy on SSD and at least my hardware distributors recommend only RAID 10 with SSDs.

RAID 5 is still the fastest for read performance if and only if the hardware raid controller cannot read from all disks in RAID 10, yet write performance is worse than RAID 10. Please consider also trying a more sophisticated benchmark for the disk performance, e.g. the flexible IO benchmark (fio).

Comparing the two different times you provided, is it possible that the RAID 5 was still in rebuild? What about the Write Back cache and its BBU settings?

LnxBil thanks for the information. RAID 5 had ended rebuild. My raid controler does not have WB nor BBU. Here is the fio results:

Code:

# fio /usr/share/doc/fio/examples/iometer-file-access-server
iometer: (groupid=0, jobs=1): err= 0: pid=3360
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3273.9MB, bw=19646KB/s, iops=6391 , runt=170637msec
    slat (usec): min=2 , max=10011 , avg= 3.68, stdev=22.35
    clat (usec): min=85 , max=42776 , avg=5816.01, stdev=4217.01
     lat (usec): min=250 , max=42779 , avg=5819.83, stdev=4216.99
    clat percentiles (usec):
     |  1.00th=[  884],  5.00th=[ 1240], 10.00th=[ 1592], 20.00th=[ 2320],
     | 30.00th=[ 3056], 40.00th=[ 3856], 50.00th=[ 4704], 60.00th=[ 5728],
     | 70.00th=[ 7008], 80.00th=[ 8768], 90.00th=[11584], 95.00th=[14144],
     | 99.00th=[19840], 99.50th=[22144], 99.90th=[27264], 99.95th=[29312],
     | 99.99th=[33536]
    bw (KB/s)  : min= 7578, max=65804, per=100.00%, avg=19656.56, stdev=12608.33
  write: io=841897KB, bw=4933.9KB/s, iops=1603 , runt=170637msec
    slat (usec): min=2 , max=11252 , avg= 5.55, stdev=74.95
    clat (msec): min=3 , max=52 , avg=16.71, stdev= 3.41
     lat (msec): min=4 , max=52 , avg=16.71, stdev= 3.41
    clat percentiles (usec):
     |  1.00th=[11200],  5.00th=[12352], 10.00th=[13120], 20.00th=[14016],
     | 30.00th=[14784], 40.00th=[15424], 50.00th=[16192], 60.00th=[17024],
     | 70.00th=[17792], 80.00th=[18816], 90.00th=[20864], 95.00th=[22912],
     | 99.00th=[28288], 99.50th=[30848], 99.90th=[36096], 99.95th=[38656],
     | 99.99th=[43776]
    bw (KB/s)  : min= 1940, max=16198, per=100.00%, avg=4936.26, stdev=3158.09
    lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.27%, 1000=1.44%
    lat (msec) : 2=10.88%, 4=20.82%, 10=34.75%, 20=28.37%, 50=3.46%
    lat (msec) : 100=0.01%
  cpu          : usr=2.54%, sys=4.59%, ctx=592425, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=1090631/w=273580/d=0, short=r=0/w=0/d=0


Run status group 0 (all jobs):
   READ: io=3273.9MB, aggrb=19646KB/s, minb=19646KB/s, maxb=19646KB/s, mint=1706                         37msec, maxt=170637msec
  WRITE: io=841897KB, aggrb=4933KB/s, minb=4933KB/s, maxb=4933KB/s, mint=170637m                         sec, maxt=170637msec


Disk stats (read/write):
    dm-0: ios=1089069/273613, merge=0/0, ticks=6311134/4563577, in_queue=1087495                         7, util=99.99%, aggrios=1089533/273672, aggrmerge=1119/331, aggrticks=6313207/45                         64746, aggrin_queue=10877812, aggrutil=99.98%
  sda: ios=1089533/273672, merge=1119/331, ticks=6313207/4564746, in_queue=10877                         812, util=99.98%

I have been dealing with lots of configurations on fstab with no changes on results.

On the other side, I have good results if I run "PassMark Performance Test" inside a VM with Windows Server 2012...

Any suggestions?
Thank you.

mir · Mar 15, 2015

If you are using ext4 as filesystem then try to add barrier=0 to your mount options.

ferranms · Mar 15, 2015

mir said:
If you are using ext4 as filesystem then try to add barrier=0 to your mount options.

Yes, ext4 and I have tried what you suggest here with no success, I've got the same results. Does not matter what config I try on fstab, always the same results... very disappointing.

Aristotlejones · Mar 15, 2015

I had the perc 310 controller and my performance with RAID 5 sucked ass too. Get rid of that piece of trash and upgrade to the expensive PERC if you want your headaches to go away. Its what I had to do.

mir · Mar 15, 2015

Do your mount options include discard?

ferranms · Mar 15, 2015

Aristotlejones said:
I had the perc 310 controller and my performance with RAID 5 sucked ass too. Get rid of that piece of trash and upgrade to the expensive PERC if you want your headaches to go away. Its what I had to do.

I'm starting to think so... but too late now.

mir, I have tried with and without discard... then I went back to defaults. This is my fstab

Code:

/dev/pve/root / ext4 errors=remount-ro 0 1
/dev/pve/data /var/lib/vz ext4 defaults 0 1
/dev/pve/swap none swap sw 0 0
proc /proc proc defaults 0 0

... and # cat proc/mounts

Code:

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,relatime,size=10240k,nr_inodes=8237456,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,noexec,relatime,size=6592248k,mode=755 0 0
/dev/mapper/pve-root / ext4 rw,relatime,errors=remount-ro,barrier=1,data=ordered 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0
tmpfs /run/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=13184480k 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/mapper/pve-data /var/lib/vz ext4 rw,relatime,barrier=1,data=ordered 0 0
rpc_pipefs /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
/dev/fuse /etc/pve fuse rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other 0 0
beancounter /proc/vz/beancounter cgroup rw,relatime,blkio,name=beancounter 0 0
container /proc/vz/container cgroup rw,relatime,freezer,devices,name=container 0 0
fairsched /proc/vz/fairsched cgroup rw,relatime,cpuacct,cpu,cpuset,name=fairsched 0 0

Maybe I'm doing something wrong... I'm really a noob in linux.

Aristotlejones · Mar 15, 2015

I wound up buckling and getting a 710. Made a huge difference. I tried just about everything under the sun and really felt hurned by dell on that purchase since they say they support Ubuntu on the server/card.

ferranms · Mar 16, 2015

Aristotlejones, unfortunatelly no chance to switch to 710. Hope that can be solved with tweaking some configs...

Aristotlejones · Mar 16, 2015

Doubt its going to work out. The card cannot keep up with what you are trying to do. Have you tried raid 1+0?

ferranms · Mar 16, 2015

No, I've only 3 HDD.

Search

Search

RAID 5 poor performance compared to RAID 1

ferranms

Active Member

m.ardito

Famous Member

kbrault

Renowned Member

LnxBil

Distinguished Member

ferranms

Active Member

mir

Famous Member

ferranms

Active Member

Aristotlejones

New Member

mir

Famous Member

ferranms

Active Member

Aristotlejones

New Member

ferranms

Active Member

Aristotlejones

New Member

ferranms

Active Member