Proxmox7 and ISCSI Storage + multipath

[starnetwork]

Hi,
I have IBM FlashSystem 5200NVMe Storage with 2 Controllers and total of 4x Network Ports, 2 from each Controller
just want to make sure we set it right and ask about the multipath settings: https://pve.proxmox.com/wiki/ISCSI_Multipath
1. I Have Added the ISCSI 4 hosts to the as Proxmox Storage
2. I checked and see all this 4 connections have same wwid by:

/lib/udev/scsi_id -g -u -d /dev/sdc
/lib/udev/scsi_id -g -u -d /dev/sdd
/lib/udev/scsi_id -g -u -d /dev/sde
/lib/udev/scsi_id -g -u -d /dev/sdf

so,
1. how can I set it to use roundrobin on multipath with same wwid in /etc/multipath.conf?
2. how can I create vg using vgcreate and map all 4 Ports to this vg for LVM mount of the Storage?
should I do vgcreate storage1 /dev/mapper/multipath-mapID ?

Regards,

 

[bbgeek17]

PVE runs on top of standard Debian OS, have you tried to look at your vendor documentation for Linux?

A quick google search for "ibm flashsystem multipath.conf" provides quiet a few documents and examples.

Side node: Your device, most likely, is not Active/Active so you will not have 4 active round-robin paths.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[starnetwork]

Thanks for that answers!
1. maybe it's can work with only 2 Active Ports, in this case round-robin should give me total of 20Gb, right?
2. my setup is correct? should I add the 4 paths of ISCSI to Proxmox as storages?
3. how can I create vg using vgcreate and map all 4 Ports to this vg for LVM mount of the Storage?
should I do vgcreate storage1 /dev/mapper/multipath-mapID ?

Regards

 

[bbgeek17]

I suspect in a proper finished configuration you will have 4 total paths, with two being active and round-robin if you wish. The other two would be standby. But I am not an expert on that particular model - vendor specific documentation on Linux/MP setup is always is always the best source.

Once you have configured the multipath you should only use the corresponding /dev/mapper/mpathx for your volume management. In fact if you try to go direct - you should get error/busy message.

The multipath will work below the Volume Manager. It (MP) will take care of any path changes and Volume Manager and anything above it will be non-the-wiser. And yes, in a perfect world you should get close to combined 20G from a storage side.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[starnetwork]

Hi,
working well, thanks!
the steps:
1. add all iSCSI ports via Proxmox GUI
2. install and set /etc/multipath.conf
3. add multipath -a wwid
4. restart multipath: systemctl restart multipath-tools.service
5. test: fio --filename=/dev/mapper/mpath0 --direct=1 --rw=read --bs=1m --size=20G --numjobs=200 --runtime=60 --group_reporting --name=file1
6. add vg manually or via GUI (in my opinion proxmox have some bug with that) and add LVM volume for /dev/mapper/mpath0

Regards,

 

[vnngoanhtuan]

Hi,

I follow the guideline at:
- https://pve.proxmox.com/wiki/Proxmox_ISCSI_installation
- https://pve.proxmox.com/wiki/ISCSI_Multipath

The status installation is ok but I do not see iSCSI target into /dev/sdX after the status login is ok and no issue.
- lsscsi no iscsi target

root@proxmox02:~# iscsiadm -m session -P 1
Target: iqn.2016-02.local.virtual:virtual.vg1 (non-flash)
Current Portal: 10.20.1.180:3260,1
Persistent Portal: 10.20.1.180:3260,1
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:674b46a9745
Iface IPaddress: 10.20.1.186
Iface HWaddress: default
Iface Netdev: default
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE

Can you help me on this issue?

Thanks,
Tuan Ngo

 

[bbgeek17]

Generally it means that you have not assigned a disk/lun to your iscsi target. Check carefully the steps for your iSCSI storage side set up.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[vnngoanhtuan]

Hi,

The same configure on Proxmox v6, the status is oke and no issues, I don't know what is happen on Proxmox v7?

You can see the image for more information.

Thanks,
Tuan Ngo

 

[bbgeek17]

Same exact target IP, target name, authentication? I.e. you are accessing one disk from both PVE6 and PVE7 at the same time with all the configuration?
Have you added the PVE7 IQN to "allowed to access" the lun in your storage? I am asking general question because you have not identified what type of iSCSI storage you are using and the configuration can be vastly different between different flavors.

So far, based on everything you said I suspect the issue is in your storage side (on iSCSI) config.

---

Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[vnngoanhtuan]

Hi,

Thanks for your reply, as my previous post, the login status is ok.

root@proxmox02:~# iscsiadm -m session -P 1
Target: iqn.2016-02.local.virtual:virtual.vg1 (non-flash)
Current Portal: 10.20.1.180:3260,1
Persistent Portal: 10.20.1.180:3260,1
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.1993-08.org.debian:01:674b46a9745
Iface IPaddress: 10.20.1.186
Iface HWaddress: default
Iface Netdev: default
SID: 1
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE

That mean:
- Same exact target IP, target name, authentication? -> are OK.
- Have you added the PVE7 IQN to "allowed to access" the lun in your storage? -> are OK.
- I.e. you are accessing one disk from both PVE6 and PVE7 at the same time with all the configuration? -> the same configure for PVE6 and PVE7.

Can you give your more idea?

 This document is work well on PVEV6 but issues on PVEV7.

We will use Multipath for link HA and improved performance. Install the needed packages first:

root@proxmox01:~# apt-get install open-iscsi multipath-tools
Then we discover the target and login:

root@proxmox01:~# systemctl start open-iscsi.service
 
root@proxmox01:~# iscsiadm -m discovery -t st -p 192.168.0.180
192.168.0.180:3260,1 iqn.2016-02.local.virtual:virtual.vg1
10.20.1.180:3260,1 iqn.2016-02.local.virtual:virtual.vg1
 
root@proxmox01:~# iscsiadm -m node --login
 
root@proxmox01:~# iscsiadm -m node
10.20.1.180:3260,1 iqn.2016-02.local.virtual:virtual.vg1
192.168.0.180:3260,1 iqn.2016-02.local.virtual:virtual.vg1
 
root@proxmox02:~# iscsiadm -m session -P 1
Target: iqn.2016-02.local.virtual:virtual.vg1 (non-flash)
    Current Portal: 10.20.1.180:3260,1
    Persistent Portal: 10.20.1.180:3260,1
        **********
        Interface:
        **********
        Iface Name: default
        Iface Transport: tcp
        Iface Initiatorname: iqn.1993-08.org.debian:01:674b46a9745
        Iface IPaddress: 10.20.1.186
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 1
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE
    Current Portal: 192.168.0.180:3260,1
    Persistent Portal: 192.168.0.180:3260,1
        **********
        Interface:
        **********
        Iface Name: default
        Iface Transport: tcp
        Iface Initiatorname: iqn.1993-08.org.debian:01:674b46a9745
        Iface IPaddress: 192.168.0.186
        Iface HWaddress: <empty>
        Iface Netdev: <empty>
        SID: 2
        iSCSI Connection State: LOGGED IN
        iSCSI Session State: LOGGED_IN
        Internal iscsid Session State: NO CHANGE
 
=> Above step are OK and NO issues

root@proxmox01:~# lsscsi
=> no list out iscsi disk on PVEV7 ; PVEV6 is OK

Thanks,
Tuan Ngo

 

[maxxis]

Hello.
You have solved your problem.
I faced the same situation.
lsscsi no list iscsi disk