iSCSI multipath with active/active dual-controller SAN

S

Stovic

New Member
Oct 17, 2025
4
0
1
Hello!

This is my first time addressing the community, and I know my issue isn't a dedicated ProxMox VE topic, but maybe someone can still help me or at least point me in the right direction. Unfortunately, the SAN manufacturer hasn't provided any information on multipath and Linux. Or I just can't find it :)

I have a question about the iSCSI multipath configuration in the following scenario (following the instructions: https://pve.proxmox.com/wiki/Multipath#Add_WWIDs_to_the_WWIDs_file):
- PVE host with 2 x 10 GbE for iSCSI (two separate VLANs/subnets)
-- nic1: 192.168.251.1
-- nic2: 192.168.252.1

- active/active dual-controller SAN with 2 x 10 GbE per controller (2 x 2) (Synology UC3200)
-- Controller-A nic1: 192.168.251.10
-- Controller-A nic2: 192.168.252.10
-- Controller-B nic1: 192.168.251.11
-- Controller-B nic2: 192.168.252.11
-- Each LUN is available on all NICs

- 2 iSCSI switches (left & right), stacked
-- Each switch operates one of the iSCSI VLANs/subnets (left: 251; right: 252)
-- The PVE host is connected to the left switch with its nic1 and to the right switch with its nic2
-- Both SAN nic1s are connected to the left switch and both SAN nic2s are connected to the right switch

This results in 4 paths per LUN. I'm aware, or rather, I believe, that only one path per VLAN/subnet (= 2 paths in total) will be actively used (isn't that one of the reasons why you should use different subnets?), but that's OK!

My actual question is: Can I safely operate this configuration? What should I be aware of? I've read about potential MAC issues?
And since I keep reading that "each path should have its own subnet": How would that even be possible with four SAN NICs and only two server NICs? Or does this always only affect the server NICs and not the SAN side?

And does it make sense to group the paths in /etc/multipath.conf because of the active/active dual controllers (a LUN is assigned to a "primary" controller, but can also be accessed by the "secondary" controller, with a performance loss -> inter-controller communication)? For example:

multipaths {
multipath {
wwid "xxxxxxxxxxxxxxxx"
alias mpath_ssd
path_grouping_policy "group_by_prio"
prio "alua"
prio_args "exclusive_pref_bit"
}
}
Or is this already (potentially unnecessary) fine-tuning?

Thank you very much,
Stefan
 
Stovic said:
I've read about potential MAC issues?
Click to expand...
Perhaps describing what you read would help to opine on this.
Stovic said:
"each path should have its own subnet"
Click to expand...
The idea is to avoid having a host (PVE) with two NICs which have IPs on the same subnet. While its possible to have a working configuration like this, most people are not able to get it right, which results in hard-to-trace errors later in the stack.

You need to keep in mind that you SAN is not really "active-active". It can serve LUN1 via Controller1, and LUN2 via Controller2. Accessing LUN1 via Controller2 would require path/LUN failover. Its not simultaneous. The technology involved is called ALUA.

Stovic said:
Or does this always only affect the server NICs and not the SAN side?
Click to expand...
presumably SAN vendors properly handle routing and source/dest replies in their "black boxes", vs you having to deal with in PVE/Linux.
Stovic said:
can also be accessed by the "secondary" controller, with a performance loss -> inter-controller communication)
Click to expand...
There will be no inter-controller communication. The system is designed to handle complete controller loss, as such the disks in the system are likely Dual-Pathed, or there is some custom Interposer involved. The LUN logically changes ownership between C1 and C2. Its somewhat seamless when it happens once. If you don't properly configure MP and bounce the LUN between controllers on every round-robin IO, there will be complaints.

Stovic said:
Unfortunately, the SAN manufacturer hasn't provided any information on multipath and Linux. Or I just can't find it
Click to expand...
the lack of documentation is not encouraging.


PS not all vendors can handle multi-homed systems. Some explicitly state that using multiple subnets is the best practice (a must). You'd need to ask your vendor for the recommendations.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Last edited:
bbgeek17 said:
Perhaps describing what you read would help opine on this.
Click to expand...
That's true, of course! Here's the link and an excerpt: https://forum.proxmox.com/threads/quick-howto-on-setting-up-iscsi-multipath.157532/
One potential problem I noticed is that, at least in my test setup, the PVE node (iSCSI initiator) responds to ARP requests for any of its IPs (10.205.81.81 and 10.205.81.91 in your example) on *both* NICs -- hence, an ARP request from the target for any of the initiator IPs gets two replies with two different MAC adresses. More specifically, on my test system, running `arping` on the target against one of the PVE IPs (172.16.0.201):
This is potentially bad, as the target may update its ARP table entries for the initiator IPs depending on which ARP request it sees first.
bbgeek17 said:
The idea is to avoid having a host (PVE) with two NICs which have IPs on the same subnet. While its possible to have a working configuration like this, most people are not able to get it right, which results in hard-to-trace errors later in the stack.
Click to expand...
So I'm probably on the safe side with 2 NICs and 2 subnets on the PVE host, no matter how many NICs the SAN has? (Unless the manufacturer says otherwise...?)!?
bbgeek17 said:
You need to keep in mind that you SAN is not really "active-active". It can serve LUN1 via Controller1, and LUN2 via Controller2. Accessing LUN1 via Controller2 would require path/LUN failover. Its not simultaneous. The technology involved is called ALUA.

presumably SAN vendors properly handle routing and source/dest replies in their "black boxes", vs you having to deal with in PVE/Linux.

There will be no inter-controller communication. The system is designed to handle complete controller loss, as such the disks in the system are likely Dual-Pathed, or there is some custom Interposer involved. The LUN logically changes ownership between C1 and C2. Its somewhat seamless when it happens once. If you don't properly configure MP and bounce the LUN between controllers on every round-robin IO, there will be complaints.
Click to expand...
Thanks for the clarification! So I should set the prioritized path in /etc/multipath.conf, all right!
bbgeek17 said:
the lack of documentation is not encouraging.
Click to expand...
!
bbgeek17 said:
PS not all vendors can handle multi-homed systems. Some explicitly state that using multiple subnets is the best practice (a must). You'd need to ask your vendor on the recommendations.
Click to expand...
I'll do that!

Thank you very much!
regards,
Stefan
 
Stovic said:
That's true, of course! Here's the link and an excerpt: https://forum.proxmox.com/threads/quick-howto-on-setting-up-iscsi-multipath.157532/
One potential problem I noticed is that, at least in my test setup, the PVE node (iSCSI initiator) responds to ARP requests for any of its IPs (10.205.81.81 and 10.205.81.91 in your example) on *both* NICs -- hence, an ARP request from the target for any of the initiator IPs gets two replies with two different MAC adresses. More specifically, on my test system, running `arping` on the target against one of the PVE IPs (172.16.0.201):
This is potentially bad, as the target may update its ARP table entries for the initiator IPs depending on which ARP request it sees first.
Click to expand...

You can also (this not a best practice in Linux but this replicates how ESXi works) change the arp_ignore value and the interface will only respond when it's directly called for. Doing this doubled the throughput on our Proxmox to Pure iSCSI multipath. Mind you, Pure is meant to do multipathing so IDK how this would work in your environment.
 
Stovic said:
This results in 4 paths per LUN. I'm aware, or rather, I believe, that only one path per VLAN/subnet (= 2 paths in total) will be actively used (isn't that one of the reasons why you should use different subnets?), but that's OK!
Click to expand...
Any of the paths can be used, but as @bbgeek17 pointed out only one controller will ACTUALLY be serving a given lun, so any traffic pointed to the other controller simply gets handed over the internal communication bus inside your SAN. This is not ideal, and your SAN documentation should give you the metrics to define MPIO so that doesnt happen accidentally; ALUA is the common method to do this so your multipath stanza looks correct to me but it would be good to check.

Stovic said:
My actual question is: Can I safely operate this configuration? What should I be aware of? I've read about potential MAC issues?
Click to expand...
Yes. as long as your NICs are on seperate subnets and not in any bonded config you shouldn't have any MAC issues. which leads to:
Stovic said:
And since I keep reading that "each path should have its own subnet": How would that even be possible with four SAN NICs and only two server NICs? Or does this always only affect the server NICs and not the SAN side?
Click to expand...
dont worry about each logical path; you have 2 actual paths on your host- nic A and nic B. as long as those are separate (one for each SAN controller) subnets you're fine. It's actually possible to use a single subnet but this way is simpler and more dependable.
 
alexskysilk said:
dont worry about each logical path; you have 2 actual paths on your host- nic A and nic B. as long as those are separate (one for each SAN controller) subnets you're fine. It's actually possible to use a single subnet but this way is simpler and more dependable.
Click to expand...
Sorry to ask, but I'm not sure what you mean by "as long as those are separate (one for each SAN controller) subnets, you're fine."
Do you think I should configure it instead of as originally mentioned:
-- Controller-A nic1: 192.168.251.10
-- Controller-A nic2: 192.168.252.10

-- Controller-B nic1: 192.168.251.11
-- Controller-B nic2: 192.168.252.11
(both subnets on both controllers)
but
-- Controller-A nic1: 192.168.251.10
-- Controller-A nic2: 192.168.251.11

-- Controller-B nic1: 192.168.252.10
-- Controller-B nic2: 192.168.252.11
(one subnet per controller)?
including changes to the wiring? (switch-A 2 times <-> controller-A & switch-B 2 times <-> controller-B)

Thanks in advance
Stefan
 
Last edited:
alexskysilk said:
you're right of course- I misspoke (miswrote?) What I mean is one vlan per physical interface on your host.
Click to expand...
Great, then I'll leave it like that, ask Synology, and play around with the path prioritization in multipath.conf! Many thanks to all contributors!
 
You must log in or register to reply here.
Top Bottom