iSCSI and dual 10GBe setup (Advice Request)

bferrell

Well-Known Member
Nov 16, 2018
99
2
48
54
So, complete Proxmox noob here. I have been reading about Proxmos, iSCSI, mulitpathing, etc. and I'm not clear on the benefits for my case.

I'm setting up 2 (for now, I will probably set these up with mirror/redundant pairs) a R620 and R720, both with dual 10G NICs. I'm also configuring a R720XD as a FreeNAS box for shared storage hosting which I've planned on using as iSCSI with ZFS since this storage can do all Proxmox storage (live migration, snapshots, dir, etc.). I have VLAN100 for the VMs and VLAN101 for the SAN. The FreeNAS will be setup with 7,200 RPM SATA spinning media.

For now, I've started configuring the proxmox nodes with the 10G NICs bonded and associated with the bridge, setup as LACP/802.3ap on VLAN100 on my UBNT switches, and 4 1G NIC ports bonded on the SAN VLAN (I had originally planned a Drobo with iSCSI and only 1G ports). After reading that I shouldn't put iSCSI on LACP, I'm not sure how to configure. I did read the section on multipathing, but I didn't understand it and it seemed fairly complicated.

Question, with 10G connections, would you expect me to see improved performance with multipathing? I kind of think not, so I'm tempted to set the nodes and the SAN with 1-10G interfaces on VLAN100 for the VMS and 1-10G nic on the SAN for storage and call it a day. Sound strategy?
 
Multipath is not (only) a question of performance improvement, it also coordinates access between the channels. If your storage has more than one IP address which can be reached from the host it's strongly recommended to use and configure it properly. Otherwise the host may see target storages twice and may become confused (even there are mechanism to detect this). For a first test you can leave it as long as the target has only one IP address (and one NIC connected).

Finally you should take care about multipath, study e.g. https://www.thegeekdiary.com/understanding-the-dm-multipath-configuration-file-etc-multipath-conf/
 
Thanks. I have a (largish but) homelab VM environment, so I can't imagine that I actually *need* multiple paths, or that I would saturate my 10G connection, which is partly why I ask the question.

I see that multipath requires they be separate networks, which seems like a big requirement, and complicated to configure properly. Until I saw that "do not use LACP" I had only planned to have the FreeNAS box on the SAN network, and to bond the dual 10G ports to this subnet with LACP (which is pretty easy). It seems unlikely that I would benefit from multipath (or get it configured properly the first try), so is it better to only use a single 10G port, or to LACP the ports? Based on what I'm reading I plan to use a single port at least until I get some experience.
 
It seems unlikely that I would benefit from multipath (or get it configured properly the first try), so is it better to only use a single 10G port, or to LACP the ports? Based on what I'm reading I plan to use a single port at least until I get some experience.


That's what I would suggest too - also LACP needs a proper configuration, and some switches may have problems with it.


Regarding multipath: the configuration is some work but not too complicate. To use it makes your setup more stable, even you use just a single channel (for the moment).
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!