Looking for a "cheap" way to upgrade to 40gibt

[justinclift]

And do I need to bring the ports on my switch up individually? The switch GUI shows that the ports with cables have cables but that they are "down"/"polling". So, do I need to "up" a port before it will connect?

Nah, it probably means your cards are in Ethernet mode rather than Infiniband mode. When you switch a card to Infiniband mode (and I think it can be done per-port) then the port in question will probably automatically show up on the switch.

It sounds like your switch has some kind of management interface too, which will probably be useful. That'll mean you probably have a subnet manager running on the switch, which is one less thing to get working.

 

[proxwolfe]

Man, this is proving so much more complicated than I anticipated (and I did anticipate issues...)!

So mstconfig tells me

Device #1:
----------

Device type:    ConnectX3Pro   
Device:         65:00.0         

Configurations:                              Next Boot
         SRIOV_EN                            True(1)         
         NUM_OF_VFS                          32             
         LOG_BAR_SIZE                        3               
         BOOT_OPTION_ROM_EN_P1               True(1)         
         BOOT_VLAN_EN_P1                     False(0)       
         BOOT_RETRY_CNT_P1                   0               
         LEGACY_BOOT_PROTOCOL_P1             PXE(1)         
         BOOT_VLAN_P1                        1               
         BOOT_OPTION_ROM_EN_P2               True(1)         
         BOOT_VLAN_EN_P2                     False(0)       
         BOOT_RETRY_CNT_P2                   0               
         LEGACY_BOOT_PROTOCOL_P2             PXE(1)         
         BOOT_VLAN_P2                        1               
         IP_VER_P1                           IPv4(0)         
         IP_VER_P2                           IPv4(0)

Note how there is no line with link_type

I am starting to suspect that my Infiniband cards don't understand Infiniband.

mstflint tells me

Image type:            FS2
FW Version:            2.43.7028
FW Release Date:       12.1.2020
Product Version:       02.43.70.28
Rom Info:              type=PXE version=3.4.662
                       type=UEFI version=14.9.90 cpu=AMD64
Device ID:             4103
Description:           Node             Port1            Port2            Sys image
GUIDs:                 ffffffffffffffff ffffffffffffffff ffffffffffffffff ffffffffffffffff
MACs:                                       e41d2ddebde0     e41d2ddebde1
VSD:                   
PSID:                  MT_1090111023

MT_1090111023 seems to refer to a ConnextX-3 Pro EN. And "EN" might mean "no Infiniband"??? Can someone who understands this, confirm either way?

I am starting to get the feeling that I would like to take my Infiniband cards out for a round of skeet

 

[justinclift]

MT_1090111023 seems to refer to a ConnextX-3 Pro EN.

Uh oh!

Yep, that's definitely one of their ethernet only models. No infiniband capability on that model of card.

So, in a 3 node cluster you'll be able to hook all three nodes up to each other when using dual port cards. But you'd need to figure something extra out in order to have a spare network port for them to communicate to anything else (ie a backup server).

Sounds like you'll be needing to get some more cards, or perhaps re-organise your architecture for the backup server?

 

[proxwolfe]

Oh FFS! How can there even be something like an IB card that can't do IB?

Anways, so does the "EN" signify cards that can't do IB? In other words, when I buy new cards, will I be safe, if I avoid "EN" cards? Is there anything else that I would want to avoid? Is there any value in buying "VPI" cards (I think my switch doesn't have the VPI profile)?

 

[justinclift]

Ahhh. I'll explain the (super simplified) history, as that should help.

Mellanox seems to have started off making Infiniband only cards (various series called "Infinihost"). After a few years of Infiniband not taking over the world they figured out how to make "dual protocol" cards which can do both Infiniband and Ethernet, a capability they call Virtual Protocol Interconnect ("VPI"). These are the ones in the ConnectX VPI, ConnectX-2 VPI, ConnectX-3 VPI, (etc) series.

At some point in the ConnectX/ConnectX-2/ConnectX-3/(etc) series Mellanox decided to also release cards that only do Ethernet. These are the Connect-X EN/ConnectX-2 EN/ConnectX-3 EN/(etc) cards.

There seem to be "Pro" models in the ConnectX-3 range too, and yours seems to be ConnectX-3 Pro EN cards. They're actually pretty good cards... if you want to do just Ethernet.

 

[justinclift]

Is there any value in buying "VPI" cards (I think my switch doesn't have the VPI profile)?

Yeah, you'll be wanting the VPI cards. Those ones can have their ports switched to Infiniband, and they'll work fine with your switch.

 

[justinclift]

Just remembered something relevant. At some point in the later ConnectX series(es) they also decided to release Infiniband only cards again.

Those ones are named a bit different too "Connect-IB". Those would probably work too (if it's a card with the right Infiniband speed), but they're generally pretty rare so you'll likely find more VPI adapters around at good prices. The VPI cards should work fine for you, though you definitely want to ensure you buy cards with the same Infiniband speed as your switch.

 

[justinclift]

@proxwolfe How'd this all end up going?