SAN Iscsi storage

[Matteo Lanzoni]

Hi, datacenter just delivered my new infrastructure and I just created cluster between 3 proxmox.
I'm doing some test to connect my shareed storage.
It is a Infortrend EONSTOR DS4042RUCB with 12x240gb SSD drives.
Datacenter gave me the following configuration for the storage:
CONTROLLER-A
-------------------------
IP1: 10.0.1.101
IP2: 10.0.1.103

CONTROLLER-B
-------------------------
IP1: 10.0.1.102
IP2: 10.0.1.104

I found in this forum that I need to use multipath-tools but i haven't experience with it.
I've read page https://pve.proxmox.com/wiki/ISCSI_Multipath but i can't find where to insert my 4 ip addresses...
Any advice?

 

[LnxBil]

You need to attach the devices directly with the "normal" iscsi admin tools, e.g. https://wiki.debian.org/SAN/iSCSI/open-iscsi

Please benchmark you solution with fio.

 

[Matteo Lanzoni]

Explain please... if I attach every ip of the SAN, i will have 4 devices.. /dev/sdc, sdd, sde, sdf.
Is this correct?

 

[LnxBil]

Yes, then you need to setup multipath, first install it, then configure it. Please refer to https://help.ubuntu.com/lts/serverguide/multipath-setting-up-dm-multipath.html

 

[Matteo Lanzoni]

Hi LnxBil, i've done fio test and the following is the result. Do you think it is good for a SAN?
Mtu value is 7980 cause SAN doesn't support a bigger one

root@pve2:~# fio --filename=/dev/mapper/mpatha --direct=1 --rw=read --bs=1m --size=20G --numjobs=200 --runtime=60 --group_reporting --name=file1
file1: (g=0): rw=read, bs=1M-1M/1M-1M/1M-1M, ioengine=sync, iodepth=1
...
fio-2.1.11
Starting 200 processes
Jobs: 200 (f=200): [R(200)] [100.0% done] [1759MB/0KB/0KB /s] [1759/0/0 iops] [eta 00m:00s]
file1: (groupid=0, jobs=200): err= 0: pid=18276: Mon Oct 10 22:32:43 2016
read : io=105140MB, bw=1749.3MB/s, iops=1749, runt= 60107msec
clat (msec): min=2, max=227, avg=114.24, stdev=59.10
lat (msec): min=2, max=227, avg=114.24, stdev=59.10
clat percentiles (msec):
| 1.00th=[ 5], 5.00th=[ 10], 10.00th=[ 21], 20.00th=[ 67],
| 30.00th=[ 76], 40.00th=[ 83], 50.00th=[ 113], 60.00th=[ 137],
| 70.00th=[ 157], 80.00th=[ 176], 90.00th=[ 196], 95.00th=[ 204],
| 99.00th=[ 212], 99.50th=[ 215], 99.90th=[ 219], 99.95th=[ 221],
| 99.99th=[ 225]
bw (KB /s): min= 4770, max=25239, per=0.50%, avg=9012.10, stdev=2369.21
lat (msec) : 4=0.38%, 10=4.70%, 20=4.85%, 50=1.69%, 100=34.16%
lat (msec) : 250=54.23%
cpu : usr=0.00%, sys=0.06%, ctx=186586, majf=0, minf=52652
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=105140/w=0/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
READ: io=105140MB, aggrb=1749.3MB/s, minb=1749.3MB/s, maxb=1749.3MB/s, mint=60107msec, maxt=60107msec

 

[LnxBil]

Thank you. Sequential Read looks very good, what about a random read test?

Here is a "normal" SSD-Test with 4K block size, QD32 and random I/O

cat > /tmp/4KQD32.fio <<EOF
[global]
# old fio versions need this
#filename=/path/to/file
time_based
runtime=30

[4kqd32_read]
description=4K QD32
blocksize=4K
rw=randread
direct=1
ioengine=libaio
iodepth=32
EOF

You can change to randrw if you do not have any data on your device. Please post your result in CODE tags for better readability.