IO Delay stats with iSCSI

T

tomstephens89

Renowned Member
Mar 10, 2014
177
7
83
Kingsclere, United Kingdom
Hi guys,

I operate a 16 node cluster with 10Gbit iSCSI through to a HP SAN as my VM storage. Performance seems excellent and I have MPIO configured on every host.

However I noticed that the IO Delay stat on the host summary page moves between 0 and 3%. Mostly 0 but hovers around upto 3% every few seconds or so.

Does this IO delay number only count the hosts own disks or does it take into consideration the iSCSI LVM disks. (Well, these would appear local). I have no performance problems and from memory a performance test I ran on the host was good. Is there a good benchmark i can run to test my iSCSI throughput?

Or is this nothing to worry about?

Tom
 
Install fio on a VM and run it with these options:
fio --description="Emulation of Intel IOmeter File Server Access Pattern" --name=iometer --bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 --rw=randrw --rwmixread=80 --direct=1 --size=4g --ioengine=libaio --iodepth=64

Size should be at least the double of the VM's available memory.
 
mir said:
Install fio on a VM and run it with these options:
fio --description="Emulation of Intel IOmeter File Server Access Pattern" --name=iometer --bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 --rw=randrw --rwmixread=80 --direct=1 --size=4g --ioengine=libaio --iodepth=64

Size should be at least the double of the VM's available memory.
Click to expand...

I am running this now. The VM's disk that I am running this on resides on a LUN that is only backed by a 4 disk RAID10 using 2TB MDL 7.2K SAS disks, so I wouldn't expect mega performance.

Heres a write test with DD:

tom@oss-flt-web01:/dev$ sudo dd if=/dev/zero of=/home/tom/test.data bs=1M count=8024;
8024+0 records in
8024+0 records out
8413773824 bytes (8.4 GB) copied, 30.4132 s, 277 MB/s

Heres a read test with DD:

tom@oss-flt-web01:/dev$ dd if=/home/tom/test.data of=/dev/null bs=1M count=8024
8024+0 records in
8024+0 records out
8413773824 bytes (8.4 GB) copied, 40.5108 s, 208 MB/s


Looks about what id expect from the MDL SAS disks backing it, considering there are another 4 VM's on that same disk set.

And here is the output of fio:

tom@oss-flt-web01:~$ sudo fio --description="Emulation of Intel IOmeter File Server Access Pattern" --name=iometer --bssplit=512/10:1k/5:2k/5:4k/60:8k/2:16k/4:32k/4:64k/10 --rw=randrw --rwmixread=80 --direct=1 --size=8g --ioengine=libaio --iodepth=64
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.3
Starting 1 process
Jobs: 1 (f=1): [m] [100.0% done] [4174KB/998KB/0KB /s] [999/246/0 iops] [eta 00m:00s]
iometer: (groupid=0, jobs=1): err= 0: pid=7592: Sat Aug 8 10:55:49 2015
Description : [Emulation of Intel IOmeter File Server Access Pattern]
read : io=6553.2MB, bw=5418.1KB/s, iops=888, runt=1238332msec
slat (usec): min=0, max=178562, avg=26.88, stdev=191.79
clat (usec): min=42, max=1657.2K, avg=70587.51, stdev=74903.31
lat (usec): min=299, max=1657.2K, avg=70615.09, stdev=74903.54
clat percentiles (usec):
| 1.00th=[ 1672], 5.00th=[ 5280], 10.00th=[ 8096], 20.00th=[13376],
| 30.00th=[19584], 40.00th=[27008], 50.00th=[37120], 60.00th=[53504],
| 70.00th=[84480], 80.00th=[136192], 90.00th=[185344], 95.00th=[220160],
| 99.00th=[305152], 99.50th=[342016], 99.90th=[428032], 99.95th=[464896],
| 99.99th=[552960]
bw (KB /s): min= 1656, max=14243, per=100.00%, avg=5424.19, stdev=1710.85
write: io=1638.1MB, bw=1355.3KB/s, iops=221, runt=1238332msec
slat (usec): min=0, max=102517, avg=33.28, stdev=303.03
clat (usec): min=44, max=1552.7K, avg=5732.59, stdev=9813.51
lat (usec): min=403, max=1552.7K, avg=5766.59, stdev=9817.28
clat percentiles (usec):
| 1.00th=[ 524], 5.00th=[ 588], 10.00th=[ 636], 20.00th=[ 716],
| 30.00th=[ 788], 40.00th=[ 908], 50.00th=[ 1896], 60.00th=[ 4080],
| 70.00th=[ 6624], 80.00th=[ 9920], 90.00th=[15168], 95.00th=[20352],
| 99.00th=[33024], 99.50th=[40704], 99.90th=[110080], 99.95th=[162816],
| 99.99th=[246784]
bw (KB /s): min= 317, max= 3609, per=100.00%, avg=1356.50, stdev=475.35
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.13%, 750=5.37%
lat (usec) : 1000=3.84%
lat (msec) : 2=1.63%, 4=3.37%, 10=12.80%, 20=16.44%, 50=23.05%
lat (msec) : 100=11.96%, 250=19.20%, 500=2.20%, 750=0.02%, 1000=0.01%
lat (msec) : 2000=0.01%
cpu : usr=2.46%, sys=7.40%, ctx=1196945, majf=0, minf=9
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued : total=r=1099710/w=274886/d=0, short=r=0/w=0/d=0


Run status group 0 (all jobs):
READ: io=6553.2MB, aggrb=5418KB/s, minb=5418KB/s, maxb=5418KB/s, mint=1238332msec, maxt=1238332msec
WRITE: io=1638.1MB, aggrb=1355KB/s, minb=1355KB/s, maxb=1355KB/s, mint=1238332msec, maxt=1238332msec


Disk stats (read/write):
vda: ios=1101349/277647, merge=0/90, ticks=77747728/1585156, in_queue=79338868, util=100.00%
 
Last edited:
mir said:
If both cache is on I can't see what else you can do.
Click to expand...

Well like I said, I am not experiencing performance problems and the performance within that VM is what id expect from a 4 disk 7.2k SAS RAID10 that is currently shared with 4 other VM's.

I was mainly wondering if that io calculation is anything to worry about or whether it even takes into consideration the added latency when using iSCSI mounted disks underneath a multipath device.
 
Performance wise I am not impressed. In my setup which is also a 4 disk RAID10 setup based on 7.2k SATA 3 disks is measure a performance which is 5 times higher than yours for both read and write IOPS.

Storage is on Omnios (Solaris derivate) using ZFS. I use SSD for log and cache.
Code: 
zpool status vMotion
  pool: vMotion
 state: ONLINE
  scan: scrub repaired 0 in 5h7m with 0 errors on Sun Aug  2 11:37:23 2015
config:

        NAME        STATE     READ WRITE CKSUM
        vMotion     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t0d0  ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0
            c2t3d0  ONLINE       0     0     0
        logs
          c5t1d0p1  ONLINE       0     0     0
        cache
          c5t1d0p2  ONLINE       0     0     0

errors: No known data errors
 
Proof (running fio while the storage was servicing 9 other VM's)
Code: 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]              
iometer: (groupid=0, jobs=1): err= 0: pid=1067: Fri Aug 14 10:16:40 2015
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3275.2MB, bw=20644KB/s, iops=3945, runt=162497msec
    slat (usec): min=7, max=154501, avg=16.45, stdev=198.28
    clat (usec): min=42, max=2859.5K, avg=13064.49, stdev=45054.72
     lat (usec): min=187, max=2859.6K, avg=13081.92, stdev=45055.57
    clat percentiles (usec):
     |  1.00th=[  258],  5.00th=[  318], 10.00th=[  370], 20.00th=[  474],
     | 30.00th=[  732], 40.00th=[ 1352], 50.00th=[ 2192], 60.00th=[ 3536],
     | 70.00th=[ 8256], 80.00th=[14528], 90.00th=[26752], 95.00th=[57600],
     | 99.00th=[171008], 99.50th=[228352], 99.90th=[403456], 99.95th=[552960],
     | 99.99th=[1531904]
    bw (KB  /s): min= 2331, max=57454, per=100.00%, avg=21002.27, stdev=9884.28
  write: io=839698KB, bw=5167.5KB/s, iops=987, runt=162497msec
    slat (usec): min=9, max=15186, avg=18.85, stdev=55.23
    clat (usec): min=186, max=2841.3K, avg=12476.86, stdev=41315.44
     lat (usec): min=208, max=2841.3K, avg=12496.68, stdev=41316.23
    clat percentiles (usec):
     |  1.00th=[  286],  5.00th=[  350], 10.00th=[  402], 20.00th=[  502],
     | 30.00th=[  700], 40.00th=[ 1240], 50.00th=[ 2064], 60.00th=[ 3184],
     | 70.00th=[ 7648], 80.00th=[14272], 90.00th=[27008], 95.00th=[54016],
     | 99.00th=[162816], 99.50th=[220160], 99.90th=[382976], 99.95th=[505856],
     | 99.99th=[1384448]
    bw (KB  /s): min=  441, max=14941, per=100.00%, avg=5256.21, stdev=2492.56
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.56%, 500=20.92%, 750=9.13%
    lat (usec) : 1000=4.71%
    lat (msec) : 2=12.76%, 4=13.96%, 10=11.13%, 20=13.03%, 50=8.21%
    lat (msec) : 100=3.05%, 250=2.16%, 500=0.33%, 750=0.03%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=6.00%, sys=17.90%, ctx=539695, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=641188/w=160445/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3275.2MB, aggrb=20644KB/s, minb=20644KB/s, maxb=20644KB/s, mint=162497msec, maxt=162497msec
  WRITE: io=839698KB, aggrb=5167KB/s, minb=5167KB/s, maxb=5167KB/s, mint=162497msec, maxt=162497msec

Disk stats (read/write):
  sda: ios=641184/160510, merge=0/33, ticks=8245336/1987552, in_queue=10261148, util=100.00%
 
mir said:
Proof (running fio while the storage was servicing 9 other VM's)
Code: 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [m(1)] [100.0% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 00m:00s]              
iometer: (groupid=0, jobs=1): err= 0: pid=1067: Fri Aug 14 10:16:40 2015
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=3275.2MB, bw=20644KB/s, iops=3945, runt=162497msec
    slat (usec): min=7, max=154501, avg=16.45, stdev=198.28
    clat (usec): min=42, max=2859.5K, avg=13064.49, stdev=45054.72
     lat (usec): min=187, max=2859.6K, avg=13081.92, stdev=45055.57
    clat percentiles (usec):
     |  1.00th=[  258],  5.00th=[  318], 10.00th=[  370], 20.00th=[  474],
     | 30.00th=[  732], 40.00th=[ 1352], 50.00th=[ 2192], 60.00th=[ 3536],
     | 70.00th=[ 8256], 80.00th=[14528], 90.00th=[26752], 95.00th=[57600],
     | 99.00th=[171008], 99.50th=[228352], 99.90th=[403456], 99.95th=[552960],
     | 99.99th=[1531904]
    bw (KB  /s): min= 2331, max=57454, per=100.00%, avg=21002.27, stdev=9884.28
  write: io=839698KB, bw=5167.5KB/s, iops=987, runt=162497msec
    slat (usec): min=9, max=15186, avg=18.85, stdev=55.23
    clat (usec): min=186, max=2841.3K, avg=12476.86, stdev=41315.44
     lat (usec): min=208, max=2841.3K, avg=12496.68, stdev=41316.23
    clat percentiles (usec):
     |  1.00th=[  286],  5.00th=[  350], 10.00th=[  402], 20.00th=[  502],
     | 30.00th=[  700], 40.00th=[ 1240], 50.00th=[ 2064], 60.00th=[ 3184],
     | 70.00th=[ 7648], 80.00th=[14272], 90.00th=[27008], 95.00th=[54016],
     | 99.00th=[162816], 99.50th=[220160], 99.90th=[382976], 99.95th=[505856],
     | 99.99th=[1384448]
    bw (KB  /s): min=  441, max=14941, per=100.00%, avg=5256.21, stdev=2492.56
    lat (usec) : 50=0.01%, 100=0.01%, 250=0.56%, 500=20.92%, 750=9.13%
    lat (usec) : 1000=4.71%
    lat (msec) : 2=12.76%, 4=13.96%, 10=11.13%, 20=13.03%, 50=8.21%
    lat (msec) : 100=3.05%, 250=2.16%, 500=0.33%, 750=0.03%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.01%
  cpu          : usr=6.00%, sys=17.90%, ctx=539695, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=641188/w=160445/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=3275.2MB, aggrb=20644KB/s, minb=20644KB/s, maxb=20644KB/s, mint=162497msec, maxt=162497msec
  WRITE: io=839698KB, aggrb=5167KB/s, minb=5167KB/s, maxb=5167KB/s, mint=162497msec, maxt=162497msec

Disk stats (read/write):
  sda: ios=641184/160510, merge=0/33, ticks=8245336/1987552, in_queue=10261148, util=100.00%
Click to expand...

Whilst that is excellent, it is clear that your IO test is almost exclusively using the cache. There is no way at all that you can achieve 3900 IOPS read with a 4 disk 7.2k SATA RAID10. Your write IOPS look more realistic but even still, with 9 other VM's using it at the same time, cache is coming into play a lot here as well I think. A new 7.2k SATA disk on average can produce in the region of 120 IOPS.

Here is another test run of mine, I need to work out what my write performance is that low, but my other VM's are database applications with frequent write loads...

Code: 
iometer: (g=0): rw=randrw, bs=512-64K/512-64K/512-64K, ioengine=libaio, iodepth=64fio-2.1.3
Starting 1 process
Jobs: 1 (f=1): [m] [100.0% done] [3988KB/863KB/0KB /s] [921/212/0 iops] [eta 00m:00s]  
iometer: (groupid=0, jobs=1): err= 0: pid=20841: Fri Aug 14 09:34:56 2015
  Description  : [Emulation of Intel IOmeter File Server Access Pattern]
  read : io=6553.2MB, bw=5339.5KB/s, iops=875, runt=1256759msec
    slat (usec): min=0, max=34141, avg=26.47, stdev=101.38
    clat (usec): min=2, max=1596.2K, avg=71704.04, stdev=77449.41
     lat (usec): min=287, max=1596.2K, avg=71731.25, stdev=77449.56
    clat percentiles (usec):
     |  1.00th=[ 1560],  5.00th=[ 5408], 10.00th=[ 8256], 20.00th=[13760],
     | 30.00th=[20096], 40.00th=[28032], 50.00th=[39168], 60.00th=[55552],
     | 70.00th=[83456], 80.00th=[132096], 90.00th=[185344], 95.00th=[226304],
     | 99.00th=[333824], 99.50th=[382976], 99.90th=[473088], 99.95th=[505856],
     | 99.99th=[585728]
    bw (KB  /s): min= 1541, max=15927, per=100.00%, avg=5344.46, stdev=1828.37
  write: io=1638.1MB, bw=1335.4KB/s, iops=218, runt=1256759msec
    slat (usec): min=0, max=25049, avg=31.15, stdev=102.32
    clat (usec): min=332, max=529360, avg=5561.77, stdev=9584.13
     lat (usec): min=378, max=529376, avg=5593.69, stdev=9584.33
    clat percentiles (usec):
     |  1.00th=[  516],  5.00th=[  588], 10.00th=[  636], 20.00th=[  716],
     | 30.00th=[  788], 40.00th=[  900], 50.00th=[ 1704], 60.00th=[ 3792],
     | 70.00th=[ 6240], 80.00th=[ 9408], 90.00th=[14656], 95.00th=[19840],
     | 99.00th=[34048], 99.50th=[42752], 99.90th=[118272], 99.95th=[171008],
     | 99.99th=[252928]
    bw (KB  /s): min=  285, max= 4594, per=100.00%, avg=1336.56, stdev=503.56
    lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.16%
    lat (usec) : 750=5.42%, 1000=3.95%
    lat (msec) : 2=1.66%, 4=3.38%, 10=12.53%, 20=15.77%, 50=22.74%
    lat (msec) : 100=13.66%, 250=18.04%, 500=2.67%, 750=0.04%, 1000=0.01%
    lat (msec) : 2000=0.01%
  cpu          : usr=2.37%, sys=7.25%, ctx=1196885, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=1099710/w=274886/d=0, short=r=0/w=0/d=0


Run status group 0 (all jobs):
   READ: io=6553.2MB, aggrb=5339KB/s, minb=5339KB/s, maxb=5339KB/s, mint=1256759msec, maxt=1256759msec
  WRITE: io=1638.1MB, aggrb=1335KB/s, minb=1335KB/s, maxb=1335KB/s, mint=1256759msec, maxt=1256759msec


Disk stats (read/write):
  vda: ios=1099719/277717, merge=0/96, ticks=78781216/1550976, in_queue=80332160, util=100.00%
 
tomstephens89 said:
Whilst that is excellent, it is clear that your IO test is almost exclusively using the cache. There is no way at all that you can achieve 3900 IOPS read with a 4 disk 7.2k SATA RAID10. Your write IOPS look more realistic but even still, with 9 other VM's using it at the same time, cache is coming into play a lot here as well I think. A new 7.2k SATA disk on average can produce in the region of 120 IOPS.
Click to expand...
Yes, of course caching play an important role here but with ZFS it is intelligent caching (I did note that SSD where used for log (write) and cache (read) caching in the pool). Another thing I forgot to mention is that my pve notes are connected to the storage via Infiniband DDR (dual 10gb) which also is a big factor to count in.

If your primary workload is databases you definitely need to improve you write performance and especially 4k write performance since databases do a lot of 4k writes.

What filesystem is on your SAN, proprietary I guess?
Have you followed the recommended configuration for database use?
 
mir said:
Yes, of course caching play an important role here but with ZFS it is intelligent caching (I did note that SSD where used for log (write) and cache (read) caching in the pool). Another thing I forgot to mention is that my pve notes are connected to the storage via Infiniband DDR (dual 10gb) which also is a big factor to count in.

If your primary workload is databases you definitely need to improve you write performance and especially 4k write performance since databases do a lot of 4k writes.

What filesystem is on your SAN, proprietary I guess?
Have you followed the recommended configuration for database use?
Click to expand...

Yeah you have a pretty quick set up there.

The database apps that i am using are running fine at the moment. I am about to add another 2 disk shelfs to the SAN which each have 25 x 10k RPM 300GB 6G SAS disks in. The DB workload will be moved to these disks eventually.

The SAN is a HP P2000 G3, with 10GB iSCSI controllers in. Each proxmox host has 2 x 10Gb links onto the storage network. Jumbo frames is enabled as is multipathing on each host.

To be honest, I think my tests indicate performance that looks about right for the hardware in use. Its not a high end SAN and the controllers only have 2GB cache + its only a 4 disk 7.2k RAID10. Factoring in the 2 write penalty, random write performance can't be far off. Sequential write gets me over 240MB/s.
 
Last edited:
tomstephens89 said:
The database apps that i am using are running fine at the moment. I am about to add another 2 disk shelfs to the SAN which each have 25 x 10k RPM 300GB 6G SAS disks in. The DB workload will be moved to these disks eventually.
Click to expand...
I would expect performance 10 times faster than now.
tomstephens89 said:
The SAN is a HP P2000 G3, with 10GB iSCSI controllers in. Each proxmox host has 2 x 10Gb links onto the storage network. Jumbo frames is enabled as is multipathing on each host.
Click to expand...
From work our old SAN was a DS4700 and DS4800 which performed ok. Now we have 4 EMC each with 2 VNX5300 controllers for our 24 ESXi 5.5 nodes connected through 8GB FC and dual filer in each for file sharing. This one performs awesome;-) (1,200,000 read IOPS and 200,000 write IOPS)
tomstephens89 said:
To be honest, I think my tests indicate performance that looks about right for the hardware in use. Its not a high end SAN and the controllers only have 2GB cache + its only a 4 disk 7.2k RAID10. Factoring in the 2 write penalty, random write performance can't be far off. Sequential write gets me over 240MB/s.
Click to expand...
I believe your are right in that assumption. In my setup there is 16GB RAM, 8 GB SSD write cache and 86GB SSD read cache.

For database sequential writes are of no importance, all that matters is random write IOPS.
 
mir said:
I would expect performance 10 times faster than now.

From work our old SAN was a DS4700 and DS4800 which performed ok. Now we have 4 EMC each with 2 VNX5300 controllers for our 24 ESXi 5.5 nodes connected through 8GB FC and dual filer in each for file sharing. This one performs awesome;-) (1,200,000 read IOPS and 200,000 write IOPS)
I believe your are right in that assumption. In my setup there is 16GB RAM, 8 GB SSD write cache and 86GB SSD read cache.

For database sequential writes are of no importance, all that matters is random write IOPS.
Click to expand...

Indeed, like I say I am not experiencing performance problems since the DB workload is currently not heavy. However I always like to know the performance of my equipment. The two new shelfs with an extra 50 10k SAS disks should get things moving a lot quicker.

I am pretty happy with my sequential read however on this small raid10.

Code: 
oss@mqttbroker:~$ dd if=/dev/zero of=here1234 bs=1G count=10 oflag=direct
10+0 records in
10+0 records out
10737418240 bytes (11 GB) copied, 39.1038 s, 275 MB/s

When I install the new disk shelves I will migrate all the existing VM disks onto it and re create the arrays differently on the 12 bay 3.5" shelf. I am thinking of create a single 10 disk RAID10 using the 2TB 7.2K DP SAS disks and leaving 2 extra for hot spares. Currently I have 2 x 4 disk RAID10's with 2 hot spares each. Its a waste of space and I was in a rush to get something online so I went with the 'quick' configuration instead of doing it properly.

I have been thinking about moving foward with storage and in particular when we out grow this MSA2000 G3. I have been looking into Ceph and it looks like a great way to build a cheaper storage solution from commodity hardware, but I don't think the performance will be anywhere near that of a good SAN. Any thoughts here? Even with 8 OSD's per server and a dedicated journal SSD + 10GB front end + 10GB replication network... Do you think you'd see good performance close to the likes of a mid range SAN?
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back