[SOLVED] Why six paths to iSCSI?

U

unsichtbarre

Member
Oct 1, 2024
55
10
8
I was experimenting with iSCSI binding for SANs which have multiple interfaces on the same network and I am not sure I understand the results. iSCSI SAN has a management interface (10.26.20.98) and two iSCSI interfaces (10.26.22.30, 10.26.22.31) and one LUN configured/presented. Unfortunately, there is no way to block this "SAN" from presenting its management interface as a potential iSCSI Target.

Network config:
Code: 
auto nic5
iface nic5 inet static
        address 10.26.22.104/24
        mtu 9000
#iSCSI 1

auto nic6
iface nic6 inet static
        address 10.26.22.124/24
        mtu 9000
#iSCSI 2

iSCSI config:
Code: 
iscsiadm -m discovery -t sendtargets -p 10.26.22.30
iscsiadm -m iface -I iscsi1 -o new
iscsiadm -m iface -I iscsi2 -o new
iscsiadm -m iface -I iscsi1 -o update -n iface.net_ifacename -v nic5
iscsiadm -m iface -I iscsi2 -o update -n iface.net_ifacename -v nic6
iscsiadm -m discovery -t sendtargets -p 10.26.22.30
iscsiadm -m node -T iqn.2010-01.com.vmsources:tsn.803a6aff2c38 -p 10.26.22.30 -I iscsi1 --login
iscsiadm -m node -T iqn.2010-01.com.vmsources:tsn.803a6aff2c38 -p 10.26.22.31 -I iscsi2 --login

Multipath setup and applied with /etc/multipath.conf

Output of iscsiadm -m session
Code: 
iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 2.1.11
Target: iqn.2010-01.com.vmsources:tsn.803a6aff2c38 (non-flash)
        Current Portal: 10.26.22.30:3260,1
        Persistent Portal: 10.26.22.30:3260,1
                **********
                Interface:
                **********
                Iface Name: iscsi1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:6a1874fae4b0
                Iface IPaddress: 10.26.22.104
                Iface HWaddress: default
                Iface Netdev: nic5
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 131072
                FirstBurstLength: 262144
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 3  State: running
                scsi3 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdg          State: running
        Current Portal: 10.26.22.31:3260,1
        Persistent Portal: 10.26.22.31:3260,1
                **********
                Interface:
                **********
                Iface Name: iscsi2
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:6a1874fae4b0
                Iface IPaddress: 10.26.22.124
                Iface HWaddress: default
                Iface Netdev: nic6
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 131072
                FirstBurstLength: 262144
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 4  State: running
                scsi4 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdh          State: running
        Current Portal: 10.26.22.30:3260,1
        Persistent Portal: 10.26.22.30:3260,1
                **********
                Interface:
                **********
                Iface Name: iscsi2
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:6a1874fae4b0
                Iface IPaddress: 10.26.22.124
                Iface HWaddress: default
                Iface Netdev: nic6
                SID: 3
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 131072
                FirstBurstLength: 262144
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 5  State: running
                scsi5 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdi          State: running
        Current Portal: 10.26.20.98:3260,1
        Persistent Portal: 10.26.20.98:3260,1
                **********
                Interface:
                **********
                Iface Name: iscsi2
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:6a1874fae4b0
                Iface IPaddress: 10.26.22.124
                Iface HWaddress: default
                Iface Netdev: nic6
                SID: 4
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 131072
                FirstBurstLength: 262144
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 6  State: running
                scsi6 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdj          State: running

                **********
                Interface:
                **********
                Iface Name: iscsi1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:6a1874fae4b0
                Iface IPaddress: 10.26.22.104
                Iface HWaddress: default
                Iface Netdev: nic5
                SID: 5
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 131072
                FirstBurstLength: 262144
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 7  State: running
                scsi7 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdk          State: running
        Current Portal: 10.26.22.31:3260,1
        Persistent Portal: 10.26.22.31:3260,1
                **********
                Interface:
                **********
                Iface Name: iscsi1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:6a1874fae4b0
                Iface IPaddress: 10.26.22.104
                Iface HWaddress: default
                Iface Netdev: nic5
                SID: 6
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 5
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 131072
                FirstBurstLength: 262144
                MaxBurstLength: 262144
                ImmediateData: No
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 8  State: running
                scsi8 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdl          State: running

There is no network route between 10.26.22.0/24 and 10.26.20.0/24, so how can it be logged in?
 
Last edited:
bbgeek17 said:
Hi @unsichtbarre ,

The only explanation , based on the technical details you posted, is that you do indeed have a route between the two networks. You should be able to figure it out with "ping" command, traceroute, and by checking ARP tables.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Tested and no route. In fact there is no gateway on 10.26.22.0/0. All I can think of is that the PVE management interface is connecting with the management interface of the SAN and falsely reporting? or iscsiadm is falsely reporting because there is no route.
 
You are the only person with full view and access to the system. You shared summary of what you think the system is and only partial snippets with us.

There are a few possible theories, you can eliminate most of them by rebooting your host and observing what the system was able to re-establish.

The session was established at the time the routing existed. No IO/access was done, so no resets occured until this point.
Someone create a host/policy routing, have you examined the routing table?
If, as you said, the SAN and PVE are on the same management subnet - that may be playing role into what you are seeing.
You have a subnet mismatch and the two subnets are L2 adjustent.

Some commands you can run:
ip addr
ip route show table all
ip rule
ss -tnp | grep 3260
ip neigh show

Or, give tcpdump a try and analyze the traffic related to two suspect sessions.

Some additional tips: https://kb.blockbridge.com/technote...nderstand-multipath-reporting-for-your-device

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: unsichtbarre
Thank you @bbgeek17 for the excellent and helpful responses! Two issues I was working with are now resolved:

Q#1: Why was 10.26.22.98 connected/LOGGED IN when not reachable on network?
Q#1A#1: Stupid primitive SAN was acting as GW to itself! None of my actual LAB SANs or virtual appliances would allow me to create a config (multiple iSCSI discovery IPs on same network) that I wanted to test, so I installed Openfiler. I have not used Openfiler in 15 years and it is a dead project, but it allowed me to do what I needed for testing.
Q#1A#2: Why would I want to test such a non-best-practices setup? Many SANs (HPE MSA, Dell, etc., none that I have available in the LAB) are currently configured this way for VMware w/Port Binding. As more and more people seek to migrate away from VMware, it is not always possible to re-configure the SANs for PVE best-practices as the VMware LUNs need to remain available during migration. Therefore, it was necessary to establish a base procedure for port-binding on PVE (subject to vendor requirements, of course).

Q#2: For me, this was equally or more important: Why was the initiator 10.26.22.104 connecting to both 10.26.22.30 and 10.26.22.31, and equally for the second initiator 10.26.22.124?
Q#1A#1: I had stupidly left /etc/iscsi/iscsid.conf node.startup set to automatic, so even though I had explicitly logged in to only the correct portals, on reboot they all got logged in to.
Q#2A#2: The solution I found (after changing /etc/iscsi/iscsid.conf node.startup set to manual) was to edit the interfaces (iscsi1, iscsi2) found in /var/lib/iscsi/nodes/<IQN>/<IP>,3260,1/ and change node.startup to automatic only for the correct nodeIP/targetIP relationships.

THX,
-JB
 
  • Like
Reactions: Onslow
You must log in or register to reply here.
Top Bottom
Back