Proxmox with iSCSI Equallogic SAN

D

DANILO MONTAGNA

Member
Jun 28, 2019
119
12
23
44
Hi,

I've setup a environment with proxmox 6 and iSCSI Equallogic (DELL), this kind of storage requires some TCP modifications to work correctly with multipath..

One feature that is necessary for this storage to improve iscsi performance is "Disable DelayACKs" on Hosts connected to the storage..

I can't find on Proxmox where can I disable delay_acks on Linux ( /proc/sys/net/ipv4/tcp_quickack ) or ( /proc/sys/net/ipv4/tcp_nodelay) ...

Without this we're seeing a lot of "TCP retransmissions" on the storage center from proxmox hosts...

How can I disable delays acks on proxmox iscsi interface nics?
 
Yep.. I know.. but I coud not find it anywhere thought linux/debian...

I'm trying to make it work right now using iscsi offload from my broadcom nic (57810).. using software iscsi (open-iscsi) I think it's not possible to disable delay ack.
 
We are observing only tcp retransmission.. above 5% in some periods.. but it was higher when the interface MTU was in 9000 .. We lowered the mtu to 1500 it dropped the tcp retransmission a little bit.. but not 100%... I'm also searching why the MTU is also causing high tcp retransmission... my entire hardware infrastructure already works great with MTU 9000 and ESXi (with delayack off).. the problem only start to happen after Proxmox deployment..
 
Last edited:
DANILO MONTAGNA said:
Yes, with multpath.. I am using multipahd!!
Click to expand...
But only on server side i assume.
So you are using group ip for both paths ?

I have only one interface (LACP bond from 2x10Gbit) on my servers...
But i really dont understand how works failover on EQL with group IP.
If i disable one port on EQL, services. What can happen ? is there a down time ?

Dell documentation is really bad for EQL products ...
 
xPakrikx said:
But only on server side i assume.
So you are using group ip for both paths ?

I have only one interface (LACP bond from 2x10Gbit) on my servers...
But i really dont understand how works failover on EQL with group IP.
If i disable one port on EQL, services. What can happen ? is there a down time ?

Dell documentation is really bad for EQL products ...
Click to expand...
EQL works as active/stb failover controller, also the firmware has a vertical port failover in case of switch down... the group iscsi IP is the central portal point of connection for a correct setup.. do not connect directly to the controller port ip.

I use group ip for multipahd, on each interface of iscsi you use the group ip, the paths are distributed using round-robin by multipathd, even with only one group ip the multipathd driver know how to connect each path to a dif iscsi iface port, dont worry about this, obviously, since you configure the multipathd correctly for this setup!

The bond method is not recommended way, but it works the same way, but you will only see a single path from EQL. your failover will be based on ethernet card, not on eql path.. But using this config, my recommendation would be not using LACP, this is not supported by iscsi and can cause downside effects.. just use a active/standby (mode 1) bond, without LACP (802.1ad)

If you put a EQL controller down, nothing happens, the GROUP IP will be migrated to the bottom/up controller.. and the path will be reconnected, this is transparent by the host side, you only need to care how to wire the cables correctly, on host side and controller side on the core switches!!! and y are good to go!

Regards,
 
Last edited:
DANILO MONTAGNA said:
EQL works as active/stb failover controller, also the firmware has a vertical port failover in case of switch down... the group iscsi IP is the central portal point of connection for a correct setup.. do not connect directly to the controller port ip.

I use group ip for multipahd, on each interface of iscsi you use the group ip, the paths are distributed using round-robin by multipathd, even with only one group ip the multipathd driver know how to connect each path to a dif iscsi iface port, dont worry about this, obviously, since you configure the multipathd correctly for this setup!

The bond method is not recommended way, but it works the same way, but you will only see a single path from EQL. your failover will be based on ethernet card, not on eql path.. But using this config, my recommendation would be not using LACP, this is not supported by iscsi and can cause downside effects.. just use a active/standby (mode 1) bond, without LACP (802.1ad)

If you put a EQL controller down, nothing happens, the GROUP IP will be migrated to the bottom/up controller.. and the path will be reconnected, this is transparent by the host side, you only need to care how to wire the cables correctly, on host side and controller side on the core switches!!! and y are good to go!

Regards,
Click to expand...
One more question about interface optimization. Did you disable LRO/GRO only on physical interfaces or on linux bridges/vlans too ? I don't know if options are heritable or not. My network topology on proxmox nodes for iSCSI is 2x10gbit<--->bond0<--->bond0.2 (so iSCSI is in vlan 2 connected thru bond0)

very thanks
 
xPakrikx said:
One more question about interface optimization. Did you disable LRO/GRO only on physical interfaces or on linux bridges/vlans too ? I don't know if options are heritable or not. My network topology on proxmox nodes for iSCSI is 2x10gbit<--->bond0<--->bond0.2 (so iSCSI is in vlan 2 connected thru bond0)

very thanks
Click to expand...

We disabled LRO/GRO only on iscsi interfaces!!! and use MTU 9000 on these interfaces!!! bridges/vlans use default MTU. To check if the options are heritable or not y can use ethtool to verify it!!
 
DANILO MONTAGNA said:
We disabled LRO/GRO only on iscsi interfaces!!! and use MTU 9000 on these interfaces!!! bridges/vlans use default MTU. To check if the options are heritable or not y can use ethtool to verify it!!
Click to expand...
We use Jumbo frames. but for us there no way to use separate fyz. NICs for iSCSI. Servers are in blade chassi (Dell m1000e) so we have only 2x10Gbit to each server (switch in m1000e is 2x Dell MXL in switch stack configuration). So thats why we are using LACP bond. So iSCSI have dedicated vlan 2 on top of that bond.

thanks for help, i try some port optimization on test node.
 
xPakrikx said:
We use Jumbo frames. but for us there no way to use separate fyz. NICs for iSCSI. Servers are in blade chassi (Dell m1000e) so we have only 2x10Gbit to each server (switch in m1000e is 2x Dell MXL in switch stack configuration). So thats why we are using LACP bond. So iSCSI have dedicated vlan 2 on top of that bond.

thanks for help, i try some port optimization on test node.
Click to expand...
Dont your nics support NIC partitioning? check this!! We also use M1000E and our nics is Qlogic 57810S, these nics support partitioning, you can allocate 8 virtualized nics to the Host OS, instead of two physical.
 
DANILO MONTAGNA said:
Dont your nics support NIC partitioning? check this!! We also use M1000E and our nics is Qlogic 57810S, these nics support partitioning, you can allocate 8 virtualized nics to the Host OS, instead of two physical.
Click to expand...
We have same NICs Qlogic 57810S in 9 nodes and 2 nodes with Intel 4P X710-k. So then every port on one interface (4 virtual ports) are connected to one 10Gbit port on MXL, every virtual interface has same vlan configuration from switch. I asume there is no support for bonding like LACP in NPAR configuration (for VMs bridge). Very interesting. I was tring to do some DCBx configuration, but we have only one EQL storage so i think there is no need for that. But NPAR looks like nice feature, thanks for tip.
 
xPakrikx said:
We have same NICs Qlogic 57810S in 9 nodes and 2 nodes with Intel 4P X710-k. So then every port on one interface (4 virtual ports) are connected to one 10Gbit port on MXL, every virtual interface has same vlan configuration from switch. I asume there is no support for bonding like LACP in NPAR configuration (for VMs bridge). Very interesting. I was tring to do some DCBx configuration, but we have only one EQL storage so i think there is no need for that. But NPAR looks like nice feature, thanks for tip.
Click to expand...
57810S an X710 support NPAR, the best setup is to use it on your environment.. y can then segregate iscsi from LAN using virtual ports... just put MXL internal ports to hosts on "trunk mode", tag all vlans y use on these ports and use two virtual NPAR ports (one from each side of MXL "A1 and A2") to create iscsi interfaces and LAN interfaces using vlan dot1q on hosts and y are good to go!!!
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom