Hello Members
I'm trying to set up iSCSI with multipath in a PVE cluster to a SAN. I want to achieve redundancy and load balancing. There are four paths to the SAN.
However, multipath only uses two paths and creates only one priority group
Nach meinen Verständnis müsste multipath zwei priority groups mit jeweils zwei Pfaden erstellen um Redundanz und Lastausgleich zu erreichen, zum Beispiel sowie hier:
PVE-Version: 9.1.4
SAN: Fujitsu DX200 S4
Each node and the SAN has two NICs, each with two ports. All VLANs are connected to and accessible via trunking on each port.
The paths:
As I understand it, I should see four devices here:
As another attempt, I mapped the iSCSI paths to specific interfaces (I had the same problems even without mapping).
I'm trying to set up iSCSI with multipath in a PVE cluster to a SAN. I want to achieve redundancy and load balancing. There are four paths to the SAN.
However, multipath only uses two paths and creates only one priority group
Code:
root@pve2:~# multipath -ll
vmstorage1 (3600000e00d2900000029175500650000) dm-0 FUJITSU,ETERNUS_DXL
size=4.0T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 9:0:0:108 sdb 8:16 active ready running
`- 11:0:0:108 sdc 8:32 active ready running
Code:
mydev1 (<wwid>) dm-1 IBM,1815 FAStT
[size=512M][features=1 queue_if_no_path][hwhandler=1 rdac]
\_ round-robin 0 [prio=6][active]
\_ 29:0:0:1 sdf 8:80 [active][ready]
\_ 28:0:1:1 sdl 8:176 [active][ready]
\_ round-robin 0 [prio=0][enabled]
\_ 28:0:0:1 sdb 8:16 [active][ghost]
\_ 29:0:1:1 sdq 65:0 [active][ghost]PVE-Version: 9.1.4
SAN: Fujitsu DX200 S4
Each node and the SAN has two NICs, each with two ports. All VLANs are connected to and accessible via trunking on each port.
The paths:
- iSCSI 1a: 10.10.10.10, iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54
- iSCSI 1b: 10.10.10.11, iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54
- iSCSI 2a: 10.10.20.10, iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54
- iSCSI 2b: 10.10.20.11, iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54
Code:
root@pve2:~# cat /etc/multipath.conf
defaults {
user_friendly_names yes
find_multipaths strict
polling_interval 5
path_selector "round-robin 0"
path_grouping_policy group_by_prio
prio alua
path_checker tur
rr_min_io 100
rr_weight priorities
failback immediate
}
blacklist {
wwid ".*"
}
blacklist_exceptions {
wwid "3600000e00d2900000029175500650000"
}
multipaths {
multipath {
wwid "3600000e00d2900000029175500650000"
alias vmstorage1
}
}
Code:
# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/3600000e00d2900000029175500650000/
Code:
root@pve2:~# iscsiadm -m session
tcp: [1] 10.10.10.11:3260,3 iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54 (non-flash)
tcp: [2] 10.10.10.10:3260,1 iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54 (non-flash)
tcp: [3] 10.10.20.11:3260,4 iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54 (non-flash)
tcp: [4] 10.10.20.10:3260,2 iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54 (non-flash)As I understand it, I should see four devices here:
Code:
root@pve2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
...
sdb 8:16 0 4T 0 disk
`-vmstorage1 252:0 0 4T 0 mpath
sdc 8:32 0 4T 0 disk
`-vmstorage1 252:0 0 4T 0 mpath
Code:
root@pve2:~# lsscsi
...
[9:0:0:108] disk FUJITSU ETERNUS_DXL 1090 /dev/sdb
[11:0:0:108] disk FUJITSU ETERNUS_DXL 1090 /dev/sdcAs another attempt, I mapped the iSCSI paths to specific interfaces (I had the same problems even without mapping).
Code:
root@pve2:~# cat /var/lib/iscsi/ifaces/iface_vlan408
# BEGIN RECORD 2.1.11
iface.iscsi_ifacename = iface_vlan408
iface.net_ifacename = eth1.408
iface.ipaddress = 10.10.10.02
iface.prefix_len = 0
iface.transport_name = tcp
iface.vlan_id = 0
iface.vlan_priority = 0
iface.iface_num = 0
iface.mtu = 0
iface.port = 0
iface.tos = 0
iface.ttl = 0
iface.tcp_wsf = 0
iface.tcp_timer_scale = 0
iface.def_task_mgmt_timeout = 0
iface.erl = 0
iface.max_receive_data_len = 0
iface.first_burst_len = 0
iface.max_outstanding_r2t = 0
iface.max_burst_len = 0
# END RECORD
root@pve2:~# cat /var/lib/iscsi/ifaces/iface_vlan409
# BEGIN RECORD 2.1.11
iface.iscsi_ifacename = iface_vlan409
iface.net_ifacename = eth3.409
iface.ipaddress = 10.10.20.02
...
# END RECORD
Code:
root@pve2:~# iscsiadm -m session -P3
iSCSI Transport Class version 2.0-870
version 2.1.11
Target: iqn.2000-09.com.fujitsu:storage-system.eternus-dxl:00295d54 (non-flash)
Current Portal: 10.10.10.11:3260,3
Persistent Portal: 10.10.10.11:3260,3
**********
Interface:
**********
Iface Name: iface_vlan408
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:pve2
Iface IPaddress: 10.10.10.02
Iface HWaddress: default
Iface Netdev: eth1.408
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 15
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 16384
FirstBurstLength: 0
MaxBurstLength: 262144
ImmediateData: No
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 8 State: running
Current Portal: 10.10.10.10:3260,1
Persistent Portal: 10.10.10.10:3260,1
**********
Interface:
**********
Iface Name: iface_vlan408
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:pve2
Iface IPaddress: 10.10.10.02
Iface HWaddress: default
Iface Netdev: eth1.408
SID: 2
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 5
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 16384
FirstBurstLength: 0
MaxBurstLength: 262144
ImmediateData: No
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 9 State: running
scsi9 Channel 00 Id 0 Lun: 108
Attached scsi disk sdb State: running
Current Portal: 10.10.20.11:3260,4
Persistent Portal: 10.10.20.11:3260,4
**********
Interface:
**********
Iface Name: iface_vlan409
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:pve2
Iface IPaddress: 10.10.20.02
Iface HWaddress: default
Iface Netdev: eth3.409
SID: 3
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 15
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 16384
FirstBurstLength: 0
MaxBurstLength: 262144
ImmediateData: No
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 10 State: running
Current Portal: 10.10.20.10:3260,2
Persistent Portal: 10.10.20.10:3260,2
**********
Interface:
**********
Iface Name: iface_vlan409
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:pve2
Iface IPaddress: 10.10.20.02
Iface HWaddress: default
Iface Netdev: eth3.409
SID: 4
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 5
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 16384
FirstBurstLength: 0
MaxBurstLength: 262144
ImmediateData: No
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 11 State: running
scsi11 Channel 00 Id 0 Lun: 108
Attached scsi disk sdc State: running