Quick HOWTO on setting up iSCSI Multipath

[PwrBank]

Spoke with Pure, they ran an `arping` from the array. Looks like there's no flapping (after the initial ping) on the array side.



Going to see if support knows of if there is any logic that prevents it from flapping.

It looks like when arping either interface, it returns the MAC for the first interface

 

[PwrBank]

Second time is reports correctly, other than the first response again. Is the first one a broadcast?

 

[PwrBank]

Any updates from any one else?

 

[fweber]

View attachment 82251

Second time is reports correctly, other than the first response again. Is the first one a broadcast?

Hi, sorry for the delay. In principle, it would be good to avoid such ARP "inconsistencies" (e.g., getting replies with different MAC addresses). In my test setup, I have seen intermittent connection issues due to these -- while the issues are to some degree mitigated by multipath, it would still be better to avoid them altogether. This should be possible by some variant of source-based routing, or using VRFs [1], but I'm still trying to find the most maintainable option.

Regarding your screenshots from posts #21 and #22: Did you change anything in the network configuration between taking those two screenshots? I'm asking because, as you point out, they look different (in #22, "most" replies report the MAC address of the correct interface, whereas in #21 most replies report the MAC address of the first interface). Are they both from the same arping invocation?

[1] https://docs.kernel.org/networking/vrf.html

 

[PwrBank]

Regarding your screenshots from posts #21 and #22: Did you change anything in the network configuration between taking those two screenshots? I'm asking because, as you point out, they look different (in #22, "most" replies report the MAC address of the correct interface, whereas in #21 most replies report the MAC address of the first interface). Are they both from the same arping invocation?

Yup, this is from the same arping with no changes between them. On Pure arrays only tech support has access to these commands and no changes were made by them or myself, so this is just how it behaves normally.

 

[bbgeek17]

The best way to remove any ambiguity is to isolate interfaces on their own VLANs/Subnets.
If that is not possible, one should carefully review vendor's best practices.

For example:
https://support.purestorage.com/bun...opics/concept/c_linux_host_configuration.html

Note: If multiple interfaces exist on the same subnet in RHEL, your iSCSI initiator may fail to connect to Pure Storage target.
In this case, you need to set sysctl's net.ipv4.conf.all.arp_ignore to 1 to force each interface to only answer ARP requests for its own
addresses. Please see RHEL KB for Issue Detail and Resolution Steps (requires Red Hat login).

The above recommendation is generally applicable to any Linux.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[PwrBank]

The best way to remove any ambiguity is to isolate interfaces on their own VLANs/Subnets.
If that is not possible, one should carefully review vendor's best practices.


The above recommendation is generally applicable to any Linux.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

How does ESXi get around this issue? It seems to be doing something similar, but I know they probably have a lot more magic behind the curtains to do so.

I will try setting the arp_ignore and report back with the results.



We can run them on different VLANs, but this is more to see if there's a technical limitation behind this or simply that it's easier to do, therefor best practice.

 

[bbgeek17]

How does ESXi get around this issue?

They were/are running proprietary "Linux like" OS and Kernel. The networking is proprietary as well.

I will try setting the arp_ignore and report back with the results.

We do not run Pure Storage, so I cannot make any recommendations or guarantees regarding the applicability of this solution.

That said, most storage vendors mention the "single subnet" use case in their documentation as a special case.
I strongly recommend extensive testing in your environment. It may initially seem like everything is configured correctly, only to find "gremlins causing unexplained behavior" a few months from now.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[PwrBank]

I strongly recommend extensive testing in your environment. It may initially seem like everything is configured correctly, only to find "gremlins causing unexplained behavior" a few months from now.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Absolutely, we have a couple of months from migration - So we're spending a lot of time getting ahead of issues.

 

[fweber]

The best way to remove any ambiguity is to isolate interfaces on their own VLANs/Subnets.
If that is not possible, one should carefully review vendor's best practices.

For example:
https://support.purestorage.com/bun...opics/concept/c_linux_host_configuration.html

Note: If multiple interfaces exist on the same subnet in RHEL, your iSCSI initiator may fail to connect to Pure Storage target.
In this case, you need to set sysctl's net.ipv4.conf.all.arp_ignore to 1 to force each interface to only answer ARP requests for its own
addresses. Please see RHEL KB for Issue Detail and Resolution Steps (requires Red Hat login).

The above recommendation is generally applicable to any Linux.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for the suggestion! I personally have not tested arp_ignore=1 yet, but it sounds like it may be a possibility.

An alternative (that allows to keep arp_ignore at its default) might be to define one VRF [1] for each path in /etc/network/interfaces and assign each iSCSI interface to its respective VRF, for example in my setup with two iSCSI interface ens19 and ens20:

auto ens19
iface ens19 inet static
    address 172.16.0.200/24
    vrf path1

auto ens20
iface ens20 inet static
    address 172.16.0.201/24
    vrf path2

auto path1
iface path1
    vrf-table auto

auto path2
iface path2
    vrf-table auto

... and then use Open-iSCSI's ifaces feature with the iSCSI interfaces ens19 and ens20 as described in the guide that was posted here.

However, please note: I've run a few tests in the setup with VRFs and so far it looks good, but these were not very realistic workloads. So @PwrBank if you are in a position to test setups, it would be interesting to hear your experience with the arp_ignore=1 and the VRF setup.

[1] https://docs.kernel.org/networking/vrf.html

 

[PwrBank]

@PwrBank if you are in a position to test setups, it would be interesting to hear your experience with the arp_ignore=1 and the VRF setup.

[1] https://docs.kernel.org/networking/vrf.html

Will do, I'll do four separate tests and report back.

1. Baseline
2. arp_ignore = 1
3. VRF
4. Both 3 & 4