iSCSI Network Considerations / Best-Practices

S

Schimmi

New Member
Dec 20, 2024
7
1
3
Hi at all,

we are actually migrating from vmware to Proxmox.
We are using a HPE MSA iSCSI-Storage. Our three-node-cluster is connected via two independent SAN-Switches. While vmware works with a technic called port-binding, there are some problems with proxmox, as we have two independent interfaces within each subnet. Afaik there is no option right now to configure all iSCSI-interfaces as a port-group which can be used with multipath.

I considered using policy based routing, which causes problems if one interface failed, as there is no active monitoring for unconfiguring the routing tables and ip-rules.
A cron-job, testing the interfaces every second seems insufficient in an 10 G performance environment.

Therefore we try to use two pairs auf lacp, which leads to the problem, that we have to bind each vlan to a different switch.
The third option would be to configure 4 different subnets on both sides, which is a heavy workload, as the storage is in production.

So, how do you solve this case? Are there any best-practices for such a scenario?
In our test-environment, we set up the lacp-option and multipathing. It works fine, but I would really apreciate using something like port-binding as there is more flexibilty and redundancy.

kind regards,
Schimmi
 
multipath should be independed of the port group thing. I'm too unfamiliar with the network side of iSCSI with port groups, yet you should see on the pve node the portals as seperate paths.
 
With port groups I’m assuming you mean LACP or bonding. That should work across both switches provided your switches support bonding across them.

Likewise VLAN should work across your LACP interfaces, because it is a higher layer in the OSI model, VLAN don’t care what kind of interface they go over.

That has nothing to do with iSCSI or multipathing which is again on a different OSI layer. Multipathing is useful for connecting to or from multiple distinct network interfaces (whether LACP or direct) on either side.

Hence why typically you can get away with multiple interfaces not being bonded at all for iSCSI traffic. Other non-iSCSI traffic may still benefit from LACP/bonding.
 
Schimmi said:
Hi at all,

we are actually migrating from vmware to Proxmox.
We are using a HPE MSA iSCSI-Storage. Our three-node-cluster is connected via two independent SAN-Switches. While vmware works with a technic called port-binding, there are some problems with proxmox, as we have two independent interfaces within each subnet. Afaik there is no option right now to configure all iSCSI-interfaces as a port-group which can be used with multipath.

I considered using policy based routing, which causes problems if one interface failed, as there is no active monitoring for unconfiguring the routing tables and ip-rules.
A cron-job, testing the interfaces every second seems insufficient in an 10 G performance environment.

Therefore we try to use two pairs auf lacp, which leads to the problem, that we have to bind each vlan to a different switch.
The third option would be to configure 4 different subnets on both sides, which is a heavy workload, as the storage is in production.

So, how do you solve this case? Are there any best-practices for such a scenario?
In our test-environment, we set up the lacp-option and multipathing. It works fine, but I would really apreciate using something like port-binding as there is more flexibilty and redundancy.

kind regards,
Schimmi
Click to expand...
In iSCSI configuration, it's will recommend to use a separated IP subnets for each iSCSI path (it likes following diagram) instead of use LACP(802.3ad). You can refernece https://pve.proxmox.com/wiki/Multipath to check how Proxmox VE's recommend on Multipath.

Code: 
|-----------|CTL1 Port.1(192.168.10.10/24) -> VLAN 10[LAN-SW1] -> NIC.1|------------|
|   iSCSI   |                                                          | Proxmox VE |
|  Storage  |                                                          |(multipathd)|
|-----------|CTL2 Port.1(192.168.11.10/24) -> VLAN 11[LAN-SW2] -> NIC.2|------------|
 
OP said that they have two independent top-of-rack switches. This means there is no single L2 domain that would allow them to span LACP bonds across the switches. OP also has two ports from each host connected to each switch, if I understand it correctly.
Generally, storage vendors recommend to use LACP for the two ports going to the same switch, and independent Subnet/VLAN for each bond. Multipath will be required for this configuration.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: Johannes S
bbgeek17 said:
OP said that they have two independent top-of-rack switches. This means there is no single L2 domain that would allow them to span LACP bonds across the switches. OP also has two ports from each host connected to each switch, if I understand it correctly.
Generally, storage vendors recommend to use LACP for the two ports going to the same switch, and independent Subnet/VLAN for each bond. Multipath will be required for this configuration.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
Correct!

Actually, our VMware-Environment ist connected via 4 independent Ports in two different subnets, which are combined with port-binding. This seems to be a VMware-thing.
LACP-Bonds accross two switches are not possible. Building a Stack is a bad option, because you can not reboot them one by one.

The best way would be to separete every link into a different subnet. But that's really hard in a running environment.
 
Best practices for one hypervisor rarely translate directly to another.

From this conversation, it sounds like you're trying to recreate port-binding functionality from ESXi to fit within your existing infrastructure constraints.

To do this, you’ll likely need to deviate from Proxmox VE best practices and manually manage iSCSI connections across hosts.

If you go down this path, I strongly recommend thorough testing and detailed documentation of your setup.

At a high level, you might consider an approach like the following:

Code: 
cat /etc/network/interfaces
# iSCSI interface 1
auto iscsi0
iface iscsi0 inet static
    address 192.168.100.101
    netmask 255.255.255.0
    mtu 9000

# iSCSI interface 2
auto iscsi1
iface iscsi1 inet static
    address 192.168.100.102
    netmask 255.255.255.0
    mtu 9000


Discover target:
iscsiadm -m discovery -t sendtargets -p <TARGET_IP>

Create binding:
iscsiadm -m iface -I iface0 -o new
iscsiadm -m iface -I iface0 -o update -n iface.net_ifacename -v iscsi0

iscsiadm -m iface -I iface1 -o new
iscsiadm -m iface -I iface1 -o update -n iface.net_ifacename -v iscsi1

Log in with binding:
# Session 1 on iscsi0
iscsiadm -m node -T <TARGET_IQN> -p <TARGET_IP> -I iface0 --login

# Session 2 on iscsi1
iscsiadm -m node -T <TARGET_IQN> -p <TARGET_IP> -I iface1 --login

Check session bindings with:
iscsiadm -m session -P 3

This approach isn’t specific to Proxmox VE, it falls under general Linux iSCSI management. There may be additional complexities or edge cases I haven’t accounted for.

If you're working with production systems, make sure HPE and PVE is supportive of your setup, or that you have a reliable vendor who can step in when it matters most.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thanks a lot for your feedback. As our production-switches support MLAG, we will switch to that option. Two LACP with two physical interfaces each and multipath should be the best option for our environment.
 
You must log in or register to reply here.
Top Bottom