New Installation on 2 HP-Proliant DL380G7 - MSAP2000G3

[Zaqen]

Hi,

I have installing Proxmox V5.0 on 2 HP proliant DL380 and i want use SAN storage (approximately 45TB)

Proxmox 1
Proxmox 2

are 2 server with proxmox (i have create cluster, add 2nd proxmo2 on proxmo1). the 2 server have 4 interface fiber channel to connecting on san.

I have create one lun of 7,3 TB to proxmox.

My 2 servers can see 4 Lun, in reality they are juste one lun, proxmox1 server and proxmo2 server, seeing 4 lun (same number and same capacity) i think because they are 4 fiber channel per server.

So, i think it's ok to work, but i don't no what must i do now to use lun and create storage.

When i m on server promox 1 or proxmox 2 i see lun on command line :

root@proxmox1# fdisk /dev/MyLun is possible, i can create partition, change type etc....


Can you help me ?

 

[menk]

i think you have to setup multipath ....

 

[menk]

https://pve.proxmox.com/wiki/ISCSI_Multipath
this is for iscsi, but i think is valid for fiber channel too.

 

[Zaqen]

thank's to your response, but not information to pve 5 configuration device on it. on pve 5, i need to create /etc/multipath.conf ?

multipath -ll is ok i see 4 disk who is on my lun

:~# multipath -ll
3600c0ff0001ae601d4d8b05a01000000 dm-2 HP,P2000 G3 FC
size=7.6T features='2 queue_if_no_path retain_attached_hw_handler' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=50 status=active
| |- 4:0:1:1 sde 8:64 active ready running
| `- 2:0:1:1 sdc 8:32 active ready running
`-+- policy='service-time 0' prio=10 status=enabled
|- 4:0:0:1 sdd 8:48 active ready running
`- 2:0:0:1 sdb 8:16 active ready running

In proxmox what must i do to use it ? I don't understand

 

[menk]

read the section "multipath configuration"; you have to configuge multipath daemon for use round robin.
so edit /etc/multipath.conf for your configuration.
this is example of my old configuration in PVE 4.x with 2 node and a SAS San Lenovo, but i don't think there is difference to pve 5.x:

defaults {
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
uid_attribute ID_SERIAL
rr_min_io 100
failback immediate
no_path_retry queue
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(td|hd)[a-z]"
devnode "^dcssblk[0-9]*"
devnode "^cciss!c[0-9]d[0-9]*"
device {
vendor "DGC"
product "LUNZ"
}
device {
vendor "EMC"
product "LUNZ"
}
device {
vendor "IBM"
product "Universal Xport"
}
device {
vendor "IBM"
product "S/390.*"
}
device {
vendor "DELL"
product "Universal Xport"
}
device {
vendor "SGI"
product "Universal Xport"
}
device {
vendor "STK"
product "Universal Xport"
}
device {
vendor "SUN"
product "Universal Xport"
}
device {
vendor "(NETAPP|LSI|ENGENIO)"
product "Universal Xport"
}
}
blacklist_exceptions {
wwid "36005076300808d10e000000000000000"
}
multipaths {
multipath {
wwid "36005076300808d10e000000000000000"
alias mpathb
}

 

[Zaqen]

Thank'you,

I do one file who copy it on 2 nodes (proxmox1 and 2) :

/etc/multipath.conf

defaults {
polling_interval 2
path_selector "round-robin 0"
path_grouping_policy multibus
getuid_callout "/lib/udev/scsi_id -g -u -d /dev/%n"
rr_min_io 100
failback immediate
no_path_retry queue
}
blacklist {
wwid .*
}

blacklist_exceptions {
wwid 3600c0ff0001ae601d4d8b05a01000000

}

devices {
device {
vendor "HP"
product "MSAP2000G3"
path_grouping_policy group_by_prio
prio rdac
path_checker rdac
path_selector "round-robin 0"
hardware_handler "1 rdac"
failback immediate
features "2 pg_init_retries 50"
no_path_retry 30
rr_min_io 100
}
}

multipaths {
multipath {
wwid 3600c0ff0001ae601d4d8b05a01000000
alias HPMSAP2000G3
}

}

In log no error when restart service (systemd start multipath.service) :


Mar 21 17:46:56 PREF33-S-PMOX1 systemd[1]: Starting Device-Mapper Multipath Device Controller...
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: path checkers start up
Mar 21 17:46:56 PREF33-S-PMOX1 systemd[1]: Started Device-Mapper Multipath Device Controller.
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: sda: using deprecated getuid callout
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: sdd: using deprecated getuid callout
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: sde: using deprecated getuid callout
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: sdb: using deprecated getuid callout
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: sdc: using deprecated getuid callout
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: HPMSAP2000G3: load table [0 16392861184 multipath 2 queue_if_no_path retain_attached_hw_handler 0 2 1 round-robin 0 2 1 8:64 1 8:32 1 round-robin 0 2 1 8:48 1 8:16 1]
Mar 21 17:46:56 PREF33-S-PMOX1 multipathd[3537]: HPMSAP2000G3: event checker started
Mar 21 17:47:00 PREF33-S-PMOX1 systemd[1]: Starting Proxmox VE replication runner...
Mar 21 17:47:01 PREF33-S-PMOX1 systemd[1]: Started Proxmox VE replication runner.

I have no error in log, but what must i do now ?

 

[menk]

now you must create lvm on top of the lun so you can use it as shared storage on the two node......
you can't use lun directly on a cluster, it isn't cluster aware.....

 

[Zaqen]

OK but where i do this, i don't understand where i can make this ? is most to make lvm thin or lvm ?

 

[menk]

from web gui of proxmox, Datacenter > storage > Add LVM ....
lvm thin can be used only for local storage, if I'm not mistaken.....
so use lvm, not lvm thin...
more info here:
https://pve.proxmox.com/wiki/LVM2

 

[Zaqen]

Hi, i don't arrive to create LVM or Isci disk on interface GUI... ;-(

 

[menk]

what's the problem???

 

[Zaqen]

I don't understand howto add Lun to create Storage.

All pictures are here : https://photos.app.goo.gl/VPSwnx1tweCNtKVo1

PMOX1 & PMOX2 seed the LUN (i have create VDISK on HP P2000 G3 FC of 8tb) it's ok

I configure multipath (configuration 1 & 2 on pictures), processus run (on pmox 1 et pmox 2) ,

multipath -ll return good status (i think)


Now, i try to add iscsi disk on proxmox gui interface, but i don't know what must i do.

San HP 2000 G3 FC, have 2 controllers (10.33.71.245 / 10.33.71.246)

multpath v3 return other information (pictures), i don't see error, what do you think ?

root@PREF33-S-PMOX1:~# multipath -v3
Apr 24 14:40:34 | libdevmapper version 1.02.137 (2016-11-30)
Apr 24 14:40:34 | DM multipath kernel driver v1.12.0
Apr 24 14:40:34 | loading //lib/multipath/libchecktur.so checker
Apr 24 14:40:34 | loading //lib/multipath/libprioconst.so prioritizer
Apr 24 14:40:34 | sda: udev property ID_WWN whitelisted
Apr 24 14:40:34 | sda: mask = 0x1f
Apr 24 14:40:34 | sda: dev_t = 8:0
Apr 24 14:40:34 | sda: size = 286677120
Apr 24 14:40:34 | sda: vendor = HP
Apr 24 14:40:34 | sda: product = LOGICAL VOLUME
Apr 24 14:40:34 | sda: rev = 3.52
Apr 24 14:40:34 | sda: h:b:t:l = 3:1:0:0
Apr 24 14:40:34 | sda: tgt_node_name =
Apr 24 14:40:34 | sda: path state = running
Apr 24 14:40:34 | sda: 17844 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:40:34 | sda: serial = 5001438009E83B30
Apr 24 14:40:34 | sda: get_state
Apr 24 14:40:34 | sda: path_checker = tur (internal default)
Apr 24 14:40:34 | sda: checker timeout = 30 ms (internal default)
Apr 24 14:40:34 | sda: state = up
Apr 24 14:40:34 | sda: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:40:34 | sda: using deprecated getuid callout
Apr 24 14:40:34 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sda
Apr 24 14:40:34 | sda: uid = 3600508b1001ca15b462cb999f65494ed (callout)
Apr 24 14:40:34 | sda: detect_prio = yes (config file default)
Apr 24 14:40:34 | sda: prio = const (internal default)
Apr 24 14:40:34 | sda: prio args = "" (internal default)
Apr 24 14:40:34 | sda: const prio = 1
Apr 24 14:40:34 | sdd: udev property ID_WWN whitelisted
Apr 24 14:40:34 | sdd: mask = 0x1f
Apr 24 14:40:34 | sdd: dev_t = 8:48
Apr 24 14:40:34 | sdd: size = 16392861184
Apr 24 14:40:34 | sdd: vendor = HP
Apr 24 14:40:34 | sdd: product = P2000 G3 FC
Apr 24 14:40:34 | sdd: rev = T252
Apr 24 14:40:34 | sdd: h:b:t:l = 4:0:0:1
Apr 24 14:40:34 | SCSI target 4:0:0 -> FC rport 4:0-0
Apr 24 14:40:34 | sdd: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:40:34 | sdd: path state = running
Apr 24 14:40:34 | sdd: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:40:34 | sdd: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:40:34 | sdd: get_state
Apr 24 14:40:34 | sdd: path_checker = tur (internal default)
Apr 24 14:40:34 | sdd: checker timeout = 30 ms (internal default)
Apr 24 14:40:34 | sdd: state = up
Apr 24 14:40:34 | sdd: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:40:34 | sdd: using deprecated getuid callout
Apr 24 14:40:34 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sdd
Apr 24 14:40:34 | sdd: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:40:34 | sdd: detect_prio = yes (config file default)
Apr 24 14:40:34 | 4:0:0:1: attribute access_state not found in sysfs
Apr 24 14:40:34 | loading //lib/multipath/libprioalua.so prioritizer
Apr 24 14:40:34 | sdd: prio = alua (detected setting)
Apr 24 14:40:34 | sdd: prio args = "" (detected setting)
Apr 24 14:40:34 | sdd: reported target port group is 1
Apr 24 14:40:34 | sdd: aas = 01 [active/non-optimized]
Apr 24 14:40:34 | sdd: alua prio = 10
Apr 24 14:40:34 | sde: udev property ID_WWN whitelisted
Apr 24 14:40:34 | sde: mask = 0x1f
Apr 24 14:40:34 | sde: dev_t = 8:64
Apr 24 14:40:34 | sde: size = 16392861184
Apr 24 14:40:34 | sde: vendor = HP
Apr 24 14:40:34 | sde: product = P2000 G3 FC
Apr 24 14:40:34 | sde: rev = T252
Apr 24 14:40:34 | sde: h:b:t:l = 4:0:1:1
Apr 24 14:40:34 | SCSI target 4:0:1 -> FC rport 4:0-1
Apr 24 14:40:34 | sde: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:40:34 | sde: path state = running
Apr 24 14:40:34 | sde: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:40:34 | sde: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:40:34 | sde: get_state
Apr 24 14:40:34 | sde: path_checker = tur (internal default)
Apr 24 14:40:34 | sde: checker timeout = 30 ms (internal default)
Apr 24 14:40:34 | sde: state = up
Apr 24 14:40:34 | sde: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:40:34 | sde: using deprecated getuid callout
Apr 24 14:40:34 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sde
Apr 24 14:40:34 | sde: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:40:34 | sde: detect_prio = yes (config file default)
Apr 24 14:40:34 | 4:0:1:1: attribute access_state not found in sysfs
Apr 24 14:40:34 | sde: prio = alua (detected setting)
Apr 24 14:40:34 | sde: prio args = "" (detected setting)
Apr 24 14:40:34 | sde: reported target port group is 0
Apr 24 14:40:34 | sde: aas = 80 [active/optimized] [preferred]
Apr 24 14:40:34 | sde: alua prio = 50
Apr 24 14:40:34 | sdb: udev property ID_WWN whitelisted
Apr 24 14:40:34 | sdb: mask = 0x1f
Apr 24 14:40:34 | sdb: dev_t = 8:16
Apr 24 14:40:34 | sdb: size = 16392861184
Apr 24 14:40:34 | sdb: vendor = HP
Apr 24 14:40:34 | sdb: product = P2000 G3 FC
Apr 24 14:40:34 | sdb: rev = T252
Apr 24 14:40:34 | sdb: h:b:t:l = 2:0:0:1
Apr 24 14:40:34 | SCSI target 2:0:0 -> FC rport 2:0-0
Apr 24 14:40:34 | sdb: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:40:34 | sdb: path state = running
Apr 24 14:40:34 | sdb: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:40:34 | sdb: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:40:34 | sdb: get_state
Apr 24 14:40:34 | sdb: path_checker = tur (internal default)
Apr 24 14:40:34 | sdb: checker timeout = 30 ms (internal default)
Apr 24 14:40:34 | sdb: state = up
Apr 24 14:40:34 | sdb: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:40:34 | sdb: using deprecated getuid callout
Apr 24 14:40:34 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sdb
Apr 24 14:40:34 | sdb: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:40:34 | sdb: detect_prio = yes (config file default)
Apr 24 14:40:34 | 2:0:0:1: attribute access_state not found in sysfs
Apr 24 14:40:34 | sdb: prio = alua (detected setting)
Apr 24 14:40:34 | sdb: prio args = "" (detected setting)
Apr 24 14:40:34 | sdb: reported target port group is 1
Apr 24 14:40:34 | sdb: aas = 01 [active/non-optimized]
Apr 24 14:40:34 | sdb: alua prio = 10
Apr 24 14:40:34 | sdc: udev property ID_WWN whitelisted
Apr 24 14:40:34 | sdc: mask = 0x1f
Apr 24 14:40:34 | sdc: dev_t = 8:32
Apr 24 14:40:34 | sdc: size = 16392861184
Apr 24 14:40:34 | sdc: vendor = HP
Apr 24 14:40:34 | sdc: product = P2000 G3 FC
Apr 24 14:40:34 | sdc: rev = T252
Apr 24 14:40:34 | sdc: h:b:t:l = 2:0:1:1
Apr 24 14:40:34 | SCSI target 2:0:1 -> FC rport 2:0-1
Apr 24 14:40:34 | sdc: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:40:34 | sdc: path state = running
Apr 24 14:40:34 | sdc: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:40:34 | sdc: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:40:34 | sdc: get_state
Apr 24 14:40:34 | sdc: path_checker = tur (internal default)
Apr 24 14:40:34 | sdc: checker timeout = 30 ms (internal default)
Apr 24 14:40:34 | sdc: state = up
Apr 24 14:40:34 | sdc: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:40:34 | sdc: using deprecated getuid callout
Apr 24 14:40:34 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sdc
Apr 24 14:40:34 | sdc: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:40:34 | sdc: detect_prio = yes (config file default)
Apr 24 14:40:34 | 2:0:1:1: attribute access_state not found in sysfs
Apr 24 14:40:34 | sdc: prio = alua (detected setting)
Apr 24 14:40:34 | sdc: prio args = "" (detected setting)
Apr 24 14:40:34 | sdc: reported target port group is 0
Apr 24 14:40:34 | sdc: aas = 80 [active/optimized] [preferred]
Apr 24 14:40:34 | sdc: alua prio = 50
Apr 24 14:40:34 | sr0: blacklisted, udev property missing
Apr 24 14:40:34 | loop0: blacklisted, udev property missing
Apr 24 14:40:34 | loop1: blacklisted, udev property missing
Apr 24 14:40:34 | loop2: blacklisted, udev property missing
Apr 24 14:40:34 | loop3: blacklisted, udev property missing
Apr 24 14:40:34 | loop4: blacklisted, udev property missing
Apr 24 14:40:34 | loop5: blacklisted, udev property missing
Apr 24 14:40:34 | loop6: blacklisted, udev property missing
Apr 24 14:40:34 | loop7: blacklisted, udev property missing
Apr 24 14:40:34 | dm-0: blacklisted, udev property missing
Apr 24 14:40:34 | dm-1: blacklisted, udev property missing
Apr 24 14:40:34 | dm-2: blacklisted, udev property missing
Apr 24 14:40:34 | dm-3: blacklisted, udev property missing
Apr 24 14:40:34 | dm-4: blacklisted, udev property missing
Apr 24 14:40:34 | dm-5: blacklisted, udev property missing
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod
3600508b1001ca15b462cb999f65494ed 3:1:0:0 sda 8:0 1 undef ready HP,LOGICA
3600c0ff0001ae601d4d8b05a01000000 4:0:0:1 sdd 8:48 10 undef ready HP,P2000
3600c0ff0001ae601d4d8b05a01000000 4:0:1:1 sde 8:64 50 undef ready HP,P2000
3600c0ff0001ae601d4d8b05a01000000 2:0:0:1 sdb 8:16 10 undef ready HP,P2000
3600c0ff0001ae601d4d8b05a01000000 2:0:1:1 sdc 8:32 50 undef ready HP,P2000
Apr 24 14:40:34 | params = 2 queue_if_no_path retain_attached_hw_handler 0 2 1 round-robin 0 2 1 8:64 1 8:32 1 round-robin 0 2 1 8:48 1 8:16 1
Apr 24 14:40:34 | status = 2 0 0 0 2 1 A 0 2 0 8:64 A 0 8:32 A 0 E 0 2 0 8:48 A 0 8:16 A 0
Apr 24 14:40:34 | HPMSAP2000G3: disassemble map [2 queue_if_no_path retain_attached_hw_handler 0 2 1 round-robin 0 2 1 8:64 1 8:32 1 round-robin 0 2 1 8:48 1 8:16 1 ]
Apr 24 14:40:34 | HPMSAP2000G3: disassemble status [2 0 0 0 2 1 A 0 2 0 8:64 A 0 8:32 A 0 E 0 2 0 8:48 A 0 8:16 A 0 ]
Apr 24 14:40:34 | sda: (HP:LOGICAL VOLUME) wwid blacklisted
Apr 24 14:40:34 | sda: orphan path, wwid blacklisted
Apr 24 14:40:34 | const prioritizer refcount 1
Apr 24 14:40:34 | tur checker refcount 5
Apr 24 14:40:34 | sdd: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:40:34 | sde: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:40:34 | sdb: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:40:34 | sdc: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:40:34 | tur checker refcount 4
Apr 24 14:40:34 | alua prioritizer refcount 4
Apr 24 14:40:34 | tur checker refcount 3
Apr 24 14:40:34 | alua prioritizer refcount 3
Apr 24 14:40:34 | tur checker refcount 2
Apr 24 14:40:34 | alua prioritizer refcount 2
Apr 24 14:40:34 | tur checker refcount 1
Apr 24 14:40:34 | alua prioritizer refcount 1
Apr 24 14:40:34 | unloading alua prioritizer
Apr 24 14:40:34 | unloading const prioritizer
Apr 24 14:40:34 | unloading tur checker
root@PREF33-S-PMOX1:~#


root@PREF33-S-PMOX2:~# multipath -v3
Apr 24 14:42:50 | libdevmapper version 1.02.137 (2016-11-30)
Apr 24 14:42:50 | DM multipath kernel driver v1.12.0
Apr 24 14:42:50 | loading //lib/multipath/libchecktur.so checker
Apr 24 14:42:50 | loading //lib/multipath/libprioconst.so prioritizer
Apr 24 14:42:50 | sda: udev property ID_WWN whitelisted
Apr 24 14:42:50 | sda: mask = 0x1f
Apr 24 14:42:50 | sda: dev_t = 8:0
Apr 24 14:42:50 | sda: size = 286677120
Apr 24 14:42:50 | sda: vendor = HP
Apr 24 14:42:50 | sda: product = LOGICAL VOLUME
Apr 24 14:42:50 | sda: rev = 3.52
Apr 24 14:42:50 | sda: h:b:t:l = 3:1:0:0
Apr 24 14:42:50 | sda: tgt_node_name =
Apr 24 14:42:50 | sda: path state = running
Apr 24 14:42:50 | sda: 17844 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:42:50 | sda: serial = 5001438009EC3AD0
Apr 24 14:42:50 | sda: get_state
Apr 24 14:42:50 | sda: path_checker = tur (internal default)
Apr 24 14:42:50 | sda: checker timeout = 30 ms (internal default)
Apr 24 14:42:50 | sda: state = up
Apr 24 14:42:50 | sda: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:42:50 | sda: using deprecated getuid callout
Apr 24 14:42:50 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sda
Apr 24 14:42:50 | sda: uid = 3600508b1001c2054c95ae4aa95dd7810 (callout)
Apr 24 14:42:50 | sda: detect_prio = yes (config file default)
Apr 24 14:42:50 | sda: prio = const (internal default)
Apr 24 14:42:50 | sda: prio args = "" (internal default)
Apr 24 14:42:50 | sda: const prio = 1
Apr 24 14:42:50 | sdd: udev property ID_WWN whitelisted
Apr 24 14:42:50 | sdd: mask = 0x1f
Apr 24 14:42:50 | sdd: dev_t = 8:48
Apr 24 14:42:50 | sdd: size = 16392861184
Apr 24 14:42:50 | sdd: vendor = HP
Apr 24 14:42:50 | sdd: product = P2000 G3 FC
Apr 24 14:42:50 | sdd: rev = T252
Apr 24 14:42:50 | sdd: h:b:t:l = 4:0:0:1
Apr 24 14:42:50 | SCSI target 4:0:0 -> FC rport 4:0-0
Apr 24 14:42:50 | sdd: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:42:50 | sdd: path state = running
Apr 24 14:42:50 | sdd: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:42:50 | sdd: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:42:50 | sdd: get_state
Apr 24 14:42:50 | sdd: path_checker = tur (internal default)
Apr 24 14:42:50 | sdd: checker timeout = 30 ms (internal default)
Apr 24 14:42:50 | sdd: state = up
Apr 24 14:42:50 | sdd: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:42:50 | sdd: using deprecated getuid callout
Apr 24 14:42:50 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sdd
Apr 24 14:42:50 | sdd: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:42:50 | sdd: detect_prio = yes (config file default)
Apr 24 14:42:50 | 4:0:0:1: attribute access_state not found in sysfs
Apr 24 14:42:50 | loading //lib/multipath/libprioalua.so prioritizer
Apr 24 14:42:50 | sdd: prio = alua (detected setting)
Apr 24 14:42:50 | sdd: prio args = "" (detected setting)
Apr 24 14:42:50 | sdd: reported target port group is 1
Apr 24 14:42:50 | sdd: aas = 01 [active/non-optimized]
Apr 24 14:42:50 | sdd: alua prio = 10
Apr 24 14:42:50 | sde: udev property ID_WWN whitelisted
Apr 24 14:42:50 | sde: mask = 0x1f
Apr 24 14:42:50 | sde: dev_t = 8:64
Apr 24 14:42:50 | sde: size = 16392861184
Apr 24 14:42:50 | sde: vendor = HP
Apr 24 14:42:50 | sde: product = P2000 G3 FC
Apr 24 14:42:50 | sde: rev = T252
Apr 24 14:42:50 | sde: h:b:t:l = 4:0:1:1
Apr 24 14:42:50 | SCSI target 4:0:1 -> FC rport 4:0-1
Apr 24 14:42:50 | sde: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:42:50 | sde: path state = running
Apr 24 14:42:50 | sde: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:42:50 | sde: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:42:50 | sde: get_state
Apr 24 14:42:50 | sde: path_checker = tur (internal default)
Apr 24 14:42:50 | sde: checker timeout = 30 ms (internal default)
Apr 24 14:42:50 | sde: state = up
Apr 24 14:42:50 | sde: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:42:50 | sde: using deprecated getuid callout
Apr 24 14:42:50 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sde
Apr 24 14:42:50 | sde: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:42:50 | sde: detect_prio = yes (config file default)
Apr 24 14:42:50 | 4:0:1:1: attribute access_state not found in sysfs
Apr 24 14:42:50 | sde: prio = alua (detected setting)
Apr 24 14:42:50 | sde: prio args = "" (detected setting)
Apr 24 14:42:50 | sde: reported target port group is 0
Apr 24 14:42:50 | sde: aas = 80 [active/optimized] [preferred]
Apr 24 14:42:50 | sde: alua prio = 50
Apr 24 14:42:50 | sdb: udev property ID_WWN whitelisted
Apr 24 14:42:50 | sdb: mask = 0x1f
Apr 24 14:42:50 | sdb: dev_t = 8:16
Apr 24 14:42:50 | sdb: size = 16392861184
Apr 24 14:42:50 | sdb: vendor = HP
Apr 24 14:42:50 | sdb: product = P2000 G3 FC
Apr 24 14:42:50 | sdb: rev = T252
Apr 24 14:42:50 | sdb: h:b:t:l = 2:0:0:1
Apr 24 14:42:50 | SCSI target 2:0:0 -> FC rport 2:0-0
Apr 24 14:42:50 | sdb: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:42:50 | sdb: path state = running
Apr 24 14:42:50 | sdb: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:42:50 | sdb: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:42:50 | sdb: get_state
Apr 24 14:42:50 | sdb: path_checker = tur (internal default)
Apr 24 14:42:50 | sdb: checker timeout = 30 ms (internal default)
Apr 24 14:42:50 | sdb: state = up
Apr 24 14:42:50 | sdb: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:42:50 | sdb: using deprecated getuid callout
Apr 24 14:42:50 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sdb
Apr 24 14:42:50 | sdb: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:42:50 | sdb: detect_prio = yes (config file default)
Apr 24 14:42:50 | 2:0:0:1: attribute access_state not found in sysfs
Apr 24 14:42:50 | sdb: prio = alua (detected setting)
Apr 24 14:42:50 | sdb: prio args = "" (detected setting)
Apr 24 14:42:50 | sdb: reported target port group is 1
Apr 24 14:42:50 | sdb: aas = 01 [active/non-optimized]
Apr 24 14:42:50 | sdb: alua prio = 10
Apr 24 14:42:50 | sdc: udev property ID_WWN whitelisted
Apr 24 14:42:50 | sdc: mask = 0x1f
Apr 24 14:42:50 | sdc: dev_t = 8:32
Apr 24 14:42:50 | sdc: size = 16392861184
Apr 24 14:42:50 | sdc: vendor = HP
Apr 24 14:42:50 | sdc: product = P2000 G3 FC
Apr 24 14:42:50 | sdc: rev = T252
Apr 24 14:42:50 | sdc: h:b:t:l = 2:0:1:1
Apr 24 14:42:50 | SCSI target 2:0:1 -> FC rport 2:0-1
Apr 24 14:42:50 | sdc: tgt_node_name = 0x208000c0ff10a940
Apr 24 14:42:50 | sdc: path state = running
Apr 24 14:42:50 | sdc: 65535 cyl, 255 heads, 63 sectors/track, start at 0
Apr 24 14:42:50 | sdc: serial = 00c0ff1ae6010000d4d8b05a01000000
Apr 24 14:42:50 | sdc: get_state
Apr 24 14:42:50 | sdc: path_checker = tur (internal default)
Apr 24 14:42:50 | sdc: checker timeout = 30 ms (internal default)
Apr 24 14:42:50 | sdc: state = up
Apr 24 14:42:50 | sdc: getuid = "/lib/udev/scsi_id -g -u -d /dev/%n" (config file default)
Apr 24 14:42:50 | sdc: using deprecated getuid callout
Apr 24 14:42:50 | formatted callout = /lib/udev/scsi_id -g -u -d /dev/sdc
Apr 24 14:42:50 | sdc: uid = 3600c0ff0001ae601d4d8b05a01000000 (callout)
Apr 24 14:42:50 | sdc: detect_prio = yes (config file default)
Apr 24 14:42:50 | 2:0:1:1: attribute access_state not found in sysfs
Apr 24 14:42:50 | sdc: prio = alua (detected setting)
Apr 24 14:42:50 | sdc: prio args = "" (detected setting)
Apr 24 14:42:50 | sdc: reported target port group is 0
Apr 24 14:42:50 | sdc: aas = 80 [active/optimized] [preferred]
Apr 24 14:42:50 | sdc: alua prio = 50
Apr 24 14:42:50 | sr0: blacklisted, udev property missing
Apr 24 14:42:50 | loop0: blacklisted, udev property missing
Apr 24 14:42:50 | loop1: blacklisted, udev property missing
Apr 24 14:42:50 | loop2: blacklisted, udev property missing
Apr 24 14:42:50 | loop3: blacklisted, udev property missing
Apr 24 14:42:50 | loop4: blacklisted, udev property missing
Apr 24 14:42:50 | loop5: blacklisted, udev property missing
Apr 24 14:42:50 | loop6: blacklisted, udev property missing
Apr 24 14:42:50 | loop7: blacklisted, udev property missing
Apr 24 14:42:50 | dm-0: blacklisted, udev property missing
Apr 24 14:42:50 | dm-1: blacklisted, udev property missing
Apr 24 14:42:50 | dm-2: blacklisted, udev property missing
Apr 24 14:42:50 | dm-3: blacklisted, udev property missing
Apr 24 14:42:50 | dm-4: blacklisted, udev property missing
Apr 24 14:42:50 | dm-5: blacklisted, udev property missing
===== paths list =====
uuid hcil dev dev_t pri dm_st chk_st vend/prod
3600508b1001c2054c95ae4aa95dd7810 3:1:0:0 sda 8:0 1 undef ready HP,LOGICA
3600c0ff0001ae601d4d8b05a01000000 4:0:0:1 sdd 8:48 10 undef ready HP,P2000
3600c0ff0001ae601d4d8b05a01000000 4:0:1:1 sde 8:64 50 undef ready HP,P2000
3600c0ff0001ae601d4d8b05a01000000 2:0:0:1 sdb 8:16 10 undef ready HP,P2000
3600c0ff0001ae601d4d8b05a01000000 2:0:1:1 sdc 8:32 50 undef ready HP,P2000
Apr 24 14:42:50 | params = 2 queue_if_no_path retain_attached_hw_handler 0 2 1 round-robin 0 2 1 8:64 1 8:32 1 round-robin 0 2 1 8:48 1 8:16 1
Apr 24 14:42:50 | status = 2 0 0 0 2 1 A 0 2 0 8:64 A 0 8:32 A 0 E 0 2 0 8:48 A 0 8:16 A 0
Apr 24 14:42:50 | HPMSAP2000G3: disassemble map [2 queue_if_no_path retain_attached_hw_handler 0 2 1 round-robin 0 2 1 8:64 1 8:32 1 round-robin 0 2 1 8:48 1 8:16 1 ]
Apr 24 14:42:50 | HPMSAP2000G3: disassemble status [2 0 0 0 2 1 A 0 2 0 8:64 A 0 8:32 A 0 E 0 2 0 8:48 A 0 8:16 A 0 ]
Apr 24 14:42:50 | sda: (HP:LOGICAL VOLUME) wwid blacklisted
Apr 24 14:42:50 | sda: orphan path, wwid blacklisted
Apr 24 14:42:50 | const prioritizer refcount 1
Apr 24 14:42:50 | tur checker refcount 5
Apr 24 14:42:50 | sdd: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:42:50 | sde: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:42:50 | sdb: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:42:50 | sdc: (HP2000 G3 FC) wwid whitelisted
Apr 24 14:42:50 | tur checker refcount 4
Apr 24 14:42:50 | alua prioritizer refcount 4
Apr 24 14:42:50 | tur checker refcount 3
Apr 24 14:42:50 | alua prioritizer refcount 3
Apr 24 14:42:50 | tur checker refcount 2
Apr 24 14:42:50 | alua prioritizer refcount 2
Apr 24 14:42:50 | tur checker refcount 1
Apr 24 14:42:50 | alua prioritizer refcount 1
Apr 24 14:42:50 | unloading alua prioritizer
Apr 24 14:42:50 | unloading const prioritizer
Apr 24 14:42:50 | unloading tur checker
root@PREF33-S-PMOX2:~#

When i try create iscsi new storage i don't know which information / parameters insert in GUI interface.

thx to your help.

Sorry to the longer message

 

[menk]

you don't have to create a iscsi LUN, you don't have a iscsi SAN, but a fibre channel.
if your multipath is correct, you have a device named HPMSAecc under /dev/mapper/ .....
thuis is the device you have to use for configure LVM......

 

[Zaqen]

thank's to your response, Menk.

I don't have this, because i dont create lvm.

is necessary use multipath with Fiber Channel ? before installing and configuring multipath, i have the 4 lun in gui interface.


root@PREF33-S-PMOX1:/dev/mapper# ls -lai
total 0
1179 drwxr-xr-x 2 root root 180 Apr 24 15:03 .
1025 drwxr-xr-x 21 root root 4520 Apr 24 15:03 ..
1180 crw------- 1 root root 10, 236 Apr 24 14:07 control
53999 lrwxrwxrwx 1 root root 7 Apr 24 15:03 mpathb -> ../dm-2
20633 lrwxrwxrwx 1 root root 7 Apr 24 14:07 pve-data -> ../dm-5
24834 lrwxrwxrwx 1 root root 7 Apr 24 14:07 pve-data_tdata -> ../dm-4
31915 lrwxrwxrwx 1 root root 7 Apr 24 14:07 pve-data_tmeta -> ../dm-3
112 lrwxrwxrwx 1 root root 7 Apr 24 14:07 pve-root -> ../dm-1
32782 lrwxrwxrwx 1 root root 7 Apr 24 14:07 pve-swap -> ../dm-0

root@PREF33-S-PMOX2:/dev/mapper# ls -lai
total 0
1179 drwxr-xr-x 2 root root 180 Apr 24 15:04 .
1025 drwxr-xr-x 21 root root 4520 Apr 24 15:04 ..
1180 crw------- 1 root root 10, 236 Apr 24 14:08 control
74550 lrwxrwxrwx 1 root root 7 Apr 24 15:04 mpathb -> ../dm-2
13874 lrwxrwxrwx 1 root root 7 Apr 24 14:08 pve-data -> ../dm-5
537 lrwxrwxrwx 1 root root 7 Apr 24 14:08 pve-data_tdata -> ../dm-4
15659 lrwxrwxrwx 1 root root 7 Apr 24 14:08 pve-data_tmeta -> ../dm-3
30748 lrwxrwxrwx 1 root root 7 Apr 24 14:08 pve-root -> ../dm-1
12257 lrwxrwxrwx 1 root root 7 Apr 24 14:08 pve-swap -> ../dm-0
root@PREF33-S-PMOX2:/dev/mapper#

i have 4 entry in /dev/ for the disk (same in interface) /dev/sdb /dev/sdc /dev/sdd /dev/sde

i must create manually (pvcreate...etc...) volume and volume group ?

I dont seeing entry to configure Fiberchannel in the Administrator Guide to proxmox v5.x...it's unreall

 

[menk]

post a lsblk ....

 

[menk]

but imho the device is /dev/mapper/dm-2 ......

 

[menk]

https://forum.proxmox.com/threads/fibre-channel-shared-storage-how.37986/
maybe this can help you.....

 

[Zaqen]

post a lsblk ....

root@PREF33-S-PMOX1:/dev# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 136.7G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 256M 0 part
└─sda3 8:3 0 136.5G 0 part
├─pve-swap 253:0 0 8G 0 lvm [SWAP]
├─pve-root 253:1 0 34G 0 lvm /
├─pve-data_tmeta 253:3 0 80M 0 lvm
│ └─pve-data 253:5 0 78.5G 0 lvm
└─pve-data_tdata 253:4 0 78.5G 0 lvm
└─pve-data 253:5 0 78.5G 0 lvm
sdb 8:16 0 7.6T 0 disk
└─mpathb 253:2 0 7.6T 0 mpath
sdc 8:32 0 7.6T 0 disk
└─mpathb 253:2 0 7.6T 0 mpath
sdd 8:48 0 7.6T 0 disk
└─mpathb 253:2 0 7.6T 0 mpath
sde 8:64 0 7.6T 0 disk
└─mpathb 253:2 0 7.6T 0 mpath
sr0 11:0 1 1024M 0 rom

 

[jerry]

I think you should do following steps:
In the console of PREF33-S-PMOX1 or PREF33-S-PMOX2:
pvcreate /dev/mapper/3600c0ff0001ae601d4d8b05a01000000
vgcreate vmstore1 /dev/mapper/3600c0ff0001ae601d4d8b05a01000000

In the PVE GUI:
Datacenter --> Storage --> Add --> LVM
Id: your friendly name eg. vmstore1
Base storage: Existing volume groups
Volume group: you should see and choose vmstore1
Enable: checkbox - selected by default
Shared: checkbox - you should select it, because your VG will be used by two hosts
Now you will see vmstore1 below local and local-lvm and you can use it as virtual machine disk(s).

According to this document there is no need to use HP Device-Mapper Multipath Enablement Kit (and create /etc/multipath.conf file) beceause of native Linux multipath is now integrated and distributed for most of the latest Linux Releases.

 

[jerry]

My 2 servers can see 4 Lun, in reality they are juste one lun, proxmox1 server and proxmo2 server, seeing 4 lun (same number and same capacity) i think because they are 4 fiber channel per server.

You're seeing four paths because servers are probably connected to the MSA not directly but via SAN switches.