SFP+ Quadport Cards Broadcom, QLogic or Intel

[dan.ger]

Hello,

we plan to upgrade the network from 10GBe Base-T to 10/25GBe SFP+/SFP28. So in the past I still prefer intel cause the are stable an run out of the box. But the Dell R740XD Server offers as QuadPort daughter board only Broadcom, QLogic and Intel, I read there are compatibility issues with switches, especially for the Dell X710VA.

We run Mellanox Connectx-6 100GBe cards without any issues for now over 2 years. So for migration and wan access the quad port cards would fit and switches available a a nice price level.

I do not want to install additional driver and no stress with the cards itself.

Any suggestion which one of the daughter boards I should use with DAC cables and switch (at the moment a ubiquiti 28 port aggregation switch is in forcus or a 26 port microtik).

 

[Falk R.]

Hi,
normally I always use Broadcom network cards. When RoCE is needed, I use Mellanox.
I have never had problems with the cards, the only network card that gave me trouble once with Proxmox was an Intel E810 100GBit.

 

[dan.ger]

Can you confirm that a Broadcom with 57414 Chipset works well? So we decided to use SFP28 with a new Mircotik Switch instead of using SFP+. Otherwise I use the Mellanox Connectx-5 CX512F Dual Port SFP28. Background is that our Netgear switch based on 10GBe Base-T consumes to much power (round about 170W/h with 22 ports Base-T and 1 SFP+ Uplink) and we try to reduce power consumption.

 

[Falk R.]

Hello,
I have the Broadcom P225P quite often in use (same chipset), also with the MikroTik CRS510 as well as CRS504 with breakout DAC

 

[dan.ger]

Thanks a lot.so would you have 4ports with the dac breakout on the 100GBe port? How much is the power consumption of that MikroTik switch?

 

[Falk R.]

Thanks a lot.so would you have 4ports with the dac breakout on the 100GBe port? How much is the power consumption of that MikroTik switch?

Yes you can connect 4 servers or ports with 25GBit each to one QSFP28 port. The switch requires about 30 watts with DAC cables. With SFPs a bit more.

What exactly do you have in mind?
I can show you a common setup.

Small 3 node PVE-Ceph cluster with P2100P 2Port 100GBit Broadcom Nics for Ceph traffic. These redundant to two CRS504 switches and 1x 100GBIt uplink to the second switch. With 3 NVMe per server I get up to 60GBit utilization and with 4NVMe per server up to 78GBit utilization.
Therefore I use an active/backup bond, because the performance is sufficient for the small cluster.

For larger clusters I prefer to use FS.com switches with more 100GBit ports.
E.g. 5 nodes with 2x S5850-48B8C switches and very large clusters with the N8560-32C switches.
I don't use 25GBit for new projects anymore, only when I upgrade older servers and there are no NVMe in use yet.
I also have very small clusters with 3 nodes and only 25 GBit, there I build a MLAG over the switches and make a LAG over the 25GBit ports.

 

[dan.ger]

Yes we have 3 Dell R740XD srever. Ceph is connected via 100Gbe Mellanox Connectx-6 dualport in routed network without a switch, that's fine. All 3 nodes have quad port Intel X550 10Gbe nics as daughterboard. 1 Bond for WAN (plugged into the switch) and one router network for migration.
We have a Netgear M7100-24 10Gbe Base-T as switch in the rack. The switch uses 12 ports an one SFP+ as uplink. The microtik only provides 10 ports - 8x SFP28 25Gbe and 2 SFP 100Gbe, So we changed the Intel X550 Quadports with Dell Broadcom 957417M4140C with SFP28 25Gbe. So we need 6 Ports on the Mikrotik switch. Another to for the Nas with dual SFP28 25Gbe. Now we are at 8 Ports. So I need at least one more port to cascade a 1Gbe switch or use 4x SFP28 to RJ-45 converters for the last 4 links.

The main issue is to reduce the power consumption of the Netgear fom 170W/h to the Mikroswitch specification max 40W/h.

 

[Falk R.]

Ok, now I understand the complete setup.
Have you ever configured a MikroTik switch? It's completely different than other switches. Also, with the CRS5xx switches, only the RouterOS is available, which has many more features, but is also much more complicated to set up, especially with complex VLAN setups.
RJ45 always needs massive power at 10 GBit, with DAC you always have the lowest power consumption. If you want to have a switch that is more like Cisco, HPE or similar, take the: fS.com S5850-24T16B or the S5850-24B and DAC cable.
If you want to practice with MikroTik, you can download the RouterOS once and test it on a VM.

 

[dan.ger]

So thanks for the hint with routerOS and VM, I bought Mellanox Connectx-4 MXC422A cards (dual port SFP+ 25Gbe) as daughterboards for the Dell Servers. It was a decision of price 15.- EUR/Card is better than 69.- EUR for one broadcom.

So now I check the MikroTik RouterOS 7.1.2. fs.com is ou of price scope and power consumption. I think a MikroTik will do, cann you send me an Screenshot of the ports with breakout cable to 4x25SFP28? Do you have DAC passive cables. cause I read that they shoul comsume less power?

 

[dan.ger]

Yes of course routerOs looks like a bit different from calssic web uis of switches, but it is ok to manage things and cli looks nice. So I think I would go with the MikroTik at all. The only thing is DAC breakthrough port configuration, for example bonding(LCAP). So if you please be so kind to send me a picture where I can see that, I be fine and buy a MikroTik switch. SwitchOs is not usable for us in the end, cause https is missing.

 

[Falk R.]

For the management of MikroTik RouterOS I always use winbox. Especially when configuring new switches, it is extremely convenient that you can also connect to the MAC instead of just IP.
I have made screenshots of my CRS504 from my Homelab. I only use 10G/40G but the configuration is identical.
Port1 is not used
Port2 has a QSFP DAC
Port3 has a fanout cable to 4x 10G, 2x 10G of which go as uplink to my other switch and two ports each to my two servers.
Port4 has a QSFP DAC

The second screenshot shows the LACP bond.

 

[dan.ger]

Perfect, thanlks a lot, so I'll go with the Mikrotik CRS510-8XS-2XQ-IN and the Mellanox Connectx-4 SFP28 25GBe Cards to reduce power consumption. So I will use 2 passive DAC 100GBe QSFP28 to 4x 25GBe SFP28 to connect the servers and 4 1GBe Transceivers to connect legacy stuff on SFP28 Ports. I hope I end around 30W/h and have enough speed on Migration and WAN network.

 

[dan.ger]

Hi Falk,

I got everything together. I connect the 100GBe breakout with one Mellanoc ConnectX-4 LX Microtik and Mellanox show link established (Mellanox Link LED lights green) but the activity LED of the mellanox card blinked cyclic green and I get not traffic over the link. I see the card with lspci and the ports detected as eno1np0 and eno2np1 on proxmox 8.0.4, so I think the driver should be loaded to use the cards. I assign one card to vmbr0 to get wan connection. The mikrotik switch is reachable on sfp28-1 WAN link (Mikrotic switch is in routed mode to get max switching performance). I put also 4 1GBE SFP transceiver in SFP28 ports and the work.

/etc/network/interfaces

auto eno1np0
iface eno1np0 inet manual

...

auto vmbr0
iface vmbr0 inet static
    address xxx.xxx.xxx.xxx/27
    gateway xxx.xxx.xxx.xxx
    bridge-ports eno1np0
    bridge-stp off
    bridge-fd 0
#Wan

ifconfig shows link but no traffic. I bought breakoutcables for Mellanox cards. Do I need to configure something in the mirkotik or on proxmox? Mikrotik switch is reseted with NO Default Config, so it is vanilla without any configuration, only address, gateway and dns servers are set. The Dell Mellanox cards are flashed to latest firmware and they are not VPI cards, they are Ethernet only Cards.

Any suggestions?

 

[dan.ger]

Solved, works now with 24W instead of 140W.

 

[radim.smehlik]

Hi. How did you solve your problem? We are facing the same issue with CRS510. We are not able to estabilish a 25G connection. Only 10G is stable. When we force ports to 25G speed, then it depends on the MK configuration, either we can see connection outages or link is estabilished without traffic.
Thanks.

 

[dan.ger]

Hi,

I add a bridge to use it as switch. I use Cisco SFP28 cables with the mikrotic and the mellanox cards . So I get instantly 25GBit links also on breakout cables.

 

[DC-CA1]

Can we now create a stack with the crs510 or other switchs

Its the most important feature missing yet..

How build a solid CEPH network without LAG between 2 switchs in STACK ...

We use hpe comware switch for now $$$

 

[Falk R.]

Can we now create a stack with the crs510 or other switchs

Its the most important feature missing yet..

How build a solid CEPH network without LAG between 2 switchs in STACK ...

We use hpe comware switch for now $$$

You can do a better Solution than a Stack.
With MikroTik Switches, you can use MLAG to create a LAG between different Switches.
With MLAG, you can update and boot Switches without a interruption.