Proxmox VE 4.1 setup with scst srpt & srptools over Infiniband

[elurex]

I don't know how many still using SRPT but I am using it as my main protocol to share LUN instead of iscsi/iser.

Package Info

root@nas:~# pveversion --verbose
proxmox-ve: 4.1-28 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-2 (running version: 4.1-2/78c5f4a2)
pve-kernel-4.2.6-1-pve: 4.2.6-28
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-29
qemu-server: 4.0-42
pve-firmware: 1.1-7
libpve-common-perl: 4.0-42
libpve-access-control: 4.0-10
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.4-18
pve-container: 1.0-35
pve-firewall: 2.0-14
pve-ha-manager: 1.0-16
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-5
lxcfs: 0.13-pve2
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve6~jessie
openvswitch-switch: 2.3.2-2

In addition, the following packages are needed

build-essential, subversion, opensm, pve-headers-4.2.6-1-pve

Then run following command, the SCST version that I use to compile is 6776

svn checkout svn://svn.code.sf.net/p/scst/svn/trunk scst-trunk.6776
cp scst-trunk.6776 scst-trunk -R
cd scst-trunk
make 2perf
make scst srpt scstadm scst_install srpt_install scstadm_install

This should compile and install scst.ko and ib_srpt.ko

Follow config will load the necessary module when starting scst service

nano /etc/init.d/scst

parse_scst_conf() {
--  SCST_MODULES="scst"
++ SCST_MODULES="scst scst_vdisk ib_srpt"

following config is a sample of /etc/scst.conf

root@nas:~# cat /etc/scst.conf
# Automatically generated by SCST Configurator v3.1.0-pre1.

HANDLER vdisk_blockio {
        DEVICE asrs_disk {
                filename /dev/zvol/rpool/vm/asrs_disk
        }

        DEVICE erp_ap {
                filename /dev/zvol/rpool/vm/erp_ap
        }
}

TARGET_DRIVER ib_srpt {
        TARGET fe80:0000:0000:0000:0002:c903:00a0:b671 {
                comp_v_mask ff
                enabled 1
                rel_tgt_id 1

                GROUP pve4 {
                        LUN 0 asrs_disk
                        LUN 1 erp_ap

                        INITIATOR *
                }
        }
}

 

[elurex]

For SRPTOOLS setup is more difficult than SRP Target. The default srptool from debian 8.1 repo does not work. Also the key component ibsrpdm -c failed to probe and return any ib device, which is very frustrating.

The work around is to use the srptools from Mellanox, the package that I use is

MLNX_OFED_LINUX-3.1-1.0.3-debian8.1-x86_64.tgz

Download it and extract it.

cd MLNX_OFED_LINUX-3.1-1.0.3-debian8.1-x86_64/DEBS/
dpkg -i libibumad*.deb
dpkg -i srptools_1.0.2-12_amd64.deb

By default ibsrpdm -c should output following info

root@san:~# ibsrpdm -c
id_ext=0002c903009f4480,ioc_guid=0002c903009f4480,dgid=fe800000000000000002c903009f4481,pkey=ffff,service_id=0002c903009f4480

However due to PVE failed to produce those info, ibstat needs to be used instead

root@san:~# ibstat
CA 'mlx4_0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.34.5000
        Hardware version: 0
        Node GUID: 0x0002c903009f4480
        System image GUID: 0x0002c903009f4483
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 2
                LMC: 0
                SM lid: 1
                Capability mask: 0x0259486a
                Port GUID: 0x0002c903009f4481
                Link layer: InfiniBand

root@san:~# ibstatus
Infiniband device 'mlx4_0' port 1 status:
  default gid:  fe80:0000:0000:0000:0002:c903:009f:4481
  base lid:  0x2
  sm lid:  0x1
  state:  4: ACTIVE
  phys state:  5: LinkUp
  rate:  40 Gb/sec (4X QDR)
  link_layer:  InfiniBand

id_ext=0002c903009f4480,ioc_guid=0002c903009f4480,dgid=fe800000000000000002c903009f4481

root@vm-ha:~/script# nano srptools.bsh
echo "id_ext=0002c903009f4480,ioc_guid=0002c903009f4480,dgid=fe800000000000000002c903009f4481" > /sys/class/infin
iband_srp/srp-mlx4_0-1/add_target

root@vm-ha:~/script# nano /etc/modules

# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

mlx4_ib
ib_umad
ib_ipoib
ib_srp

root@vm-ha:~/script# nano /etc/rc.local

/root/script/srptools.bsh

exit 0

Then when you lsscsi, you should see your SRPT disk like following

root@vm-ha:~/script# lsscsi
[0:0:1:0]    cd/dvd  MATSHITA DVD-ROM SR-8178  PZ16  /dev/sr0
[2:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sda
[3:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdb
[4:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdc
[5:0:0:0]    disk    ATA      WDC WD40EFRX-68W 0A82  /dev/sdd
[6:0:0:0]    disk    ATA      INTEL SSDSC2BW12 RG21  /dev/sde
[9:0:0:0]    disk    SCST_BIO asrs_disk         320  /dev/sdf
[9:0:0:1]    disk    SCST_BIO erp_ap            320  /dev/sdg
[9:0:0:2]    disk    SCST_BIO erp_kac           320  /dev/sdh
[9:0:0:3]    disk    SCST_BIO mail_cfm          320  /dev/sdi
[9:0:0:4]    disk    SCST_BIO mail_disk         320  /dev/sdj
[9:0:0:5]    disk    SCST_BIO mail_data         320  /dev/sdk
[9:0:0:6]    disk    SCST_BIO oracle            320  /dev/sdl
[9:0:0:7]    disk    SCST_BIO oracle_data       320  /dev/sdm
[9:0:0:8]    disk    SCST_BIO report_disk       320  /dev/sdn