iSCSI and Multiple NICs

[Cutthroat]

How should this be setup?
I have an iSCSI SAN with 4 - 10GB NICs in the same IP subnet.
I have PVE hosts with 2 - 10GB NICs to be used for iSCSI communication to the SAN.
I have installed and configured multipath.
Right now, I have the two NICs in the same IP subnet.
I am seeing multiple paths to the storage and things appear to be working fine.

For various reasons, I tried to set my Migration network to use the iSCSI IP subnet/NICs.
Migration wouldn't work and would error out because it has two IP's in the same subnet.

I read that using LACP with iSCSI isn't recommended. But, if I was to use LACP, I would only have one IP for the two NIC's so I wouldn't have the two IP/subnet problem.
I think the other option would have to be to have two different IP subnets for iSCSI? This would provide the multipath, and have a different IP subnet for each NIC, thus no duplicate IPs in same subnet.
The issue is the SAN is configured for only one subnet, which works great in vSphere setup, but I am finding maybe not so much so with PVE setups.
Is anything going to break or not work from an iSCSI perspective having the NICs being on the same IP subnet? I think I can rework a configuration to fix the Migration network issue using some other NICs instead of the iSCSI ones. Yes, coming from a vSphere background with Network Port Binding.
Any suggestions on any other designs?

Thanks

 

[bbgeek17]

Migration wouldn't work and would error out because it has two IP's in the same subnet.

How did you come to this particular conclusion? Perhaps you can share your migration settings, how its executed, and what errors you received?

I read that using LACP with iSCSI isn't recommended.

I would disagree with this statement.

I think the other option would have to be to have two different IP subnets for iSCSI?

Generally, this is a recommended way - VMware strongly preferred it this way. Not all systems and administrators out there are capable of propermulti-home operations. Dual networks make things simpler.

hus no duplicate IPs in same subnet.

You should never have DUPLICATE IPs. Even if you multihome a system (have two NICs on the same subnet, i.e. 192.168.100.10 and 192.168.100.11) - the IPs should never collide.

but I am finding maybe not so much so with PVE setups.

Based on your previous statement, it seems you do not have your network properly configured. It is certainly possible to have PVE+iSCSI with Multipath on a single VLAN.

Is anything going to break or not work from an iSCSI perspective having the NICs being on the same IP subnet?

Not from iSCSI perspective.

Any suggestions on any other designs?

Either Multihome, or LACP, or dual-Subnet are all acceptable solutions.

This article may be of help: https://kb.blockbridge.com/technote/proxmox-lvm-shared-storage/

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Cutthroat]

How did you come to this particular conclusion? Perhaps you can share your migration settings, how its executed, and what errors you received?

I came to this because it didn't work. I don't have the exact error message handy as I had to change it back to default. Basically it didn't know which NIC to use to talk to the remote host because there were two IP's on the same subnet. I'll see if I can get a screenshot.

I would disagree with this statement.

It had to do with proper NIC utilization and even distribution of traffic.

You should never have DUPLICATE IPs. Even if you multihome a system (have two NICs on the same subnet, i.e. 192.168.100.10 and 192.168.100.11) - the IPs should never collide.

I don't have the same IP, each NIC has a unique IP, but the same IP subnet is used for both NICs.

Based on your previous statement, it seems you do not have your network properly configured. It is certainly possible to have PVE+iSCSI with Multipath on a single VLAN.

Like I said, I am seeing no issues with my setup from an iSCSI perspective, just when I try to use those NICs and the configuration for Migration purposes.

The catalyst to this whole issue is Corosync. The management IP for the hosts are on a pair of 10GB nics in a LACP Bond. I have a dedicated 1GB NIC for Corosync, but it is in the same IP subnet as the management IP. When Migration is set to Default, it is using the 1GB corosync network instead of the 10GB LACP Bond, but since the management and Corosync are on the same subnet I can't tell it which network to use as they are the same. So I was trying to find a way to use the iSCSI NICs so I didn't have to break Corosync and cluster communication trying to change their IP addresses.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[bbgeek17]

It had to do with proper NIC utilization and even distribution of traffic.

All achievable depending on the many other variables in the infrastructure.

I don't have the same IP, each NIC has a unique IP, but the same IP subnet is used for both NICs.

Understood, I was confused by this statement:

This would provide the multipath, and have a different IP subnet for each NIC, thus no duplicate IPs in same subnet.

It is a bit hard to get a full picture from the bits and pieces you provided over several messages. Especially given that you did not share you actual configuration state, only human descriptions. I would say the easiest method is to add a secondary IP outside your shared subnet to the interface you want to use, then you can set the migration networks to it:
pvesh set /cluster/options --migration_network 10.10.30.0/24

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox