iSCSI multipath with multiple nics on server and storage

mitjax

Member
Jan 5, 2020
4
0
21
47
Hi,

I am playing arround with Proxmox 6.0-9, trying to figure out, how multipath works. I did not found any explicit answers, so I am asking here.
My configuration is as follows:
  • Lenovo Storage DE2000H (Dual controller with dual ports)
  • 2x Lenovo Servers with 4port nics each
Following multiple guides including https://pve.proxmox.com/wiki/ISCSI_Multipath I am still missing something to correctly do my config.

My setup is as following:
Server1, nic1: 172.16.1.2/29
Server1, nic2: 172.16.1.10/29
Server2, nic1: 172.16.1.18/29
Server2, nic2: 172.16.1.26/29

As you can see, every nic is in its own broadcast domain. I can ping my storage from all broadcast domains as every port on storage is on every broadcast domain (basically I have 4 separate networks).

Adding disks to datacenter seems trivial with GUI: Datacenter -> Storage -> Add -> iSCSI: Name, portal IP, Target
As the brodcast domain is limited to only once nic on server, this newly added storage is not reachable by Server2.
What are the correct steps to have multipath working?


Thank you.


Code:
defaults {
        polling_interval        2
        path_selector           "round-robin 0"
        path_grouping_policy    multibus
        uid_attribute           ID_SERIAL
        rr_min_io               100
        failback                immediate
        no_path_retry           queue
        user_friendly_names     yes
}

blacklist {
     devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
     devnode "^hd[a-z][[0-9]*]"
     devnode "^sda[[0-9]*]"
     devnode "^cciss!c[0-9]d[0-9]*"
}

multipaths {
    multipath {
        wwid    36d039ea0000146e7000003005d6304f3
        alias   disk1
    }
    multipath {
        wwid    36d039ea000014601000003a45d64f8ee
        alias   disk2
    }
    multipath {
        wwid    36d039ea0000146e7000007465d64f962
        alias   disk3
    }
    multipath {
        wwid    36d039ea000014601000003a85d64f935
        alias   disk4
    }
    multipath {
        wwid    36d039ea0000146010000524a5dad9a06
        alias   disk5
    }
    multipath {
        wwid    36d039ea0000146e7000034365dadaba1
        alias   disk6
    }
    multipath {
        wwid    36d039ea00001460100005f855dae4ac2
        alias   disk7
    }
    multipath {
        wwid    36d039ea0000146e70000347b5db4bf0f
        alias   disk8
    }
    multipath {
        wwid    36d039ea0000146010000611d5db4c3fc
        alias   disk9
    }
    multipath {
        wwid    36d039ea000014601000001ea5d6261eb
        alias   disk10
    }
}
 
good morning, check this if help:

https://forum.proxmox.com/threads/use-isci-on-specific-nics.62549/

Some advices:

Server1, nic1: Use bond (set for vmbr0 VM's, corosync Cluster HA)
Server1, nic2: Use bond (set for vmbr0 VM's, corosync Cluster HA)

- Storage Area Network:
Server1, nic3: 172.16.1.10/24 (Set for dedicated storage use in multipath)
Server1, nic4: 172.16.1.10/24 (Set for dedicated storage use in multipath)

Server2, nic1: Use bond (set for vmbr0 VM's, corosync Cluster HA)
Server2, nic2: Use bond (set for vmbr0 VM's, corosync Cluster HA)

- Storage Area Network:
Server2, nic1: 172.16.1.18/29 (Set for dedicated storage use in multipath)
Server2, nic2: 172.16.1.26/29 (Set for dedicated storage use in multipath)

Example:
Obs:
vmbr1 in this case, is for some VM's to access and mount some storage NFS in this SAN network:

Captura de Tela 2020-01-06 às 09.17.57.png
 
Hi @mplssilva

Your server1 nic3 & nic4 are from same broadcast domain or this is just a typo?
Other than that, your config looks like mine.

The topology from the link you sent is the same as in my network. Following further the github link...if you check my multipath output it says there are multiple path to same destination.

The part describing volume creation from github seems like the important one...looks like the preferred way is to create LVM on iscsi mapped device.

What I don't understand is if I select the steps: Proxmox -> Datacenter -> Storage -> Add -> iscsi -> target IP
Once the path is mounted it works just fine. But when I try redundancy, disabling "target IP" on storage, the access to the path is not active anymore. So there is no redundacy. Is there a way to make it highly available other than use LVM on iscsi?

Code:
root@proxmox1:~# multipath -ll
disk1 (36d039ea0000146e7000007465d64f962) dm-5 LENOVO,DE_Series
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:4  sdk  8:160  active ready running
  |- 22:0:0:4  sdan 66:112 active ready running
  |- 20:0:0:4  sdo  8:224  active ready running
  `- 21:0:0:4  sdak 66:64  active ready running
disk2 (36d039ea000014601000003a45d64f8ee) dm-9 LENOVO,DE_Series
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:3  sdh  8:112  active ready running
  |- 22:0:0:3  sdal 66:80  active ready running
  |- 20:0:0:3  sdl  8:176  active ready running
  `- 21:0:0:3  sdai 66:32  active ready running
disk3 (36d039ea000014601000001ea5d6261eb) dm-8 LENOVO,DE_Series
size=20G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:1  sdc  8:32   active ready running
  |- 22:0:0:1  sdah 66:16  active ready running
  |- 20:0:0:1  sdf  8:80   active ready running
  `- 21:0:0:1  sdaf 65:240 active ready running
disk4 (36d039ea0000146e7000003005d6304f3) dm-14 LENOVO,DE_Series
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:2  sde  8:64   active ready running
  |- 22:0:0:2  sdaj 66:48  active ready running
  |- 20:0:0:2  sdi  8:128  active ready running
  `- 21:0:0:2  sdag 66:0   active ready running
disk5 (36d039ea000014601000003a85d64f935) dm-10 LENOVO,DE_Series
size=200G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:5  sdn  8:208  active ready running
  |- 22:0:0:5  sdap 66:144 active ready running
  |- 20:0:0:5  sdr  65:16  active ready running
  `- 21:0:0:5  sdam 66:96  active ready running
 
Hi @mplssilva

Your server1 nic3 & nic4 are from same broadcast domain or this is just a typo?
Other than that, your config looks like mine.

The topology from the link you sent is the same as in my network. Following further the github link...if you check my multipath output it says there are multiple path to same destination.

The part describing volume creation from github seems like the important one...looks like the preferred way is to create LVM on iscsi mapped device.

What I don't understand is if I select the steps: Proxmox -> Datacenter -> Storage -> Add -> iscsi -> target IP
Once the path is mounted it works just fine. But when I try redundancy, disabling "target IP" on storage, the access to the path is not active anymore. So there is no redundacy. Is there a way to make it highly available other than use LVM on iscsi?

Code:
root@proxmox1:~# multipath -ll
disk1 (36d039ea0000146e7000007465d64f962) dm-5 LENOVO,DE_Series
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:4  sdk  8:160  active ready running
  |- 22:0:0:4  sdan 66:112 active ready running
  |- 20:0:0:4  sdo  8:224  active ready running
  `- 21:0:0:4  sdak 66:64  active ready running
disk2 (36d039ea000014601000003a45d64f8ee) dm-9 LENOVO,DE_Series
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:3  sdh  8:112  active ready running
  |- 22:0:0:3  sdal 66:80  active ready running
  |- 20:0:0:3  sdl  8:176  active ready running
  `- 21:0:0:3  sdai 66:32  active ready running
disk3 (36d039ea000014601000001ea5d6261eb) dm-8 LENOVO,DE_Series
size=20G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:1  sdc  8:32   active ready running
  |- 22:0:0:1  sdah 66:16  active ready running
  |- 20:0:0:1  sdf  8:80   active ready running
  `- 21:0:0:1  sdaf 65:240 active ready running
disk4 (36d039ea0000146e7000003005d6304f3) dm-14 LENOVO,DE_Series
size=100G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:2  sde  8:64   active ready running
  |- 22:0:0:2  sdaj 66:48  active ready running
  |- 20:0:0:2  sdi  8:128  active ready running
  `- 21:0:0:2  sdag 66:0   active ready running
disk5 (36d039ea000014601000003a85d64f935) dm-10 LENOVO,DE_Series
size=200G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=30 status=active
  |- 19:0:0:5  sdn  8:208  active ready running
  |- 22:0:0:5  sdap 66:144 active ready running
  |- 20:0:0:5  sdr  65:16  active ready running
  `- 21:0:0:5  sdam 66:96  active ready running

Sorry, wrong type (In storage IP and Mask):

Server1, nic1: Use bond (set for vmbr0 VM's, corosync Cluster HA)
Server1, nic2: Use bond (set for vmbr0 VM's, corosync Cluster HA)

- Storage Area Network:
Server1, nic3: 172.16.1.10/24 (Set for dedicated storage use in multipath)
Server1, nic4: 172.16.2.11/24 (Set for dedicated storage use in multipath)

Server2, nic1: Use bond (set for vmbr0 VM's, corosync Cluster HA)
Server2, nic2: Use bond (set for vmbr0 VM's, corosync Cluster HA)

- Storage Area Network:
Server2, nic3: 172.16.1.18/24 (Set for dedicated storage use in multipath)
Server2, nic4: 172.16.2.19/24 (Set for dedicated storage use in multipath)

- Initiator (Server1 node console):
iscsiadm -m node -l -T <target name> -p 172.16.1.99 (example IP Storage - nic1)
iscsiadm -m node -l -T <target name> -p 172.16.2.99 (example IP Storage - nic2)

iscsiadm -m session -P1 (to view current sessions and verify static IP NICs are connecting to correct targets.)

lsblk (should show you recently attached targets.)
lib/udev/scsi_id -g -d /dev/sdX (to get wwid)

set /etc/multipath.conf (wwid from above)

systemctl restart multipath-tools

multipath -ll (this should display the alias and the two paths.)

** Repeat steps above for Server2 node console

- Return Server1 node console
dmsetup ls

pvcreate /dev/mapper/Storage1
vgcreate iscsi_Storage1 /dev/mapper/Storage1

- Log back into PVE web GUI
Datacenter | Sorage | Add | LVM
ID: Storage1
Volume Group: iscsi_Storage1
 
Last edited:
Basically you are using the same broadcast domain over nic3 and nic4 on both servers. How do you manage to get rid of ARP flux issues? With arp_announce and arp_ignore?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!