ConnectX5 100Gbps Problem with cards using latest drivers

[tane]

Hi I am trying to setup ConnectX5 cards.
in Mesh config using DAC cables

                    ┌───────────┐
                    │   Node1   │
                    ├─────┬─────┤
                    │enp197s0f0np0│enp197s0f1np1│
                    └──┬──┴──┬──┘
                       │     │
┌───────┬─────┐        │     │        ┌─────┬───────┐
│       │enp197s0f0np0├────────┘     └────────┤enp197s0f1np1│       │
│ Node2 ├─────┤                       ├─────┤ Node3 │
│       │enp197s0f1np1├───────────────────────┤enp197s0f0np0│       │
└───────┴─────┘                       └─────┴───────┘

So far I installed the drivers and DOCA 2.8.0 drivers
Everything looks ok

I setup the basic ip to ping hosts

Node1 
auto enp197s0f0np0
iface enp197s0f0np0 inet static
       address 10.5.5.3
       netmask 255.255.255.0

auto enp197s0f1np1
iface enp197s0f1np1 inet static
       address 10.5.5.4
       netmask 255.255.255.0

Node2

auto enp197s0f0np0
iface enp197s0f0np0 inet static
       address 10.5.5.6
       netmask 255.255.255.0

auto enp197s0f1np1
iface enp197s0f1np1 inet static
       address 10.5.5.9
       netmask 255.255.255.0

Node3
auto enp197s0f0np0
iface enp197s0f0np0 inet static
       address 10.5.5.7
       netmask 255.255.255.0

auto enp197s0f1np1
iface enp197s0f1np1 inet static
       address 10.5.5.8
       netmask 255.255.255.0

My cards are

 Device Type:      ConnectX5
  Part Number:      MCX516A-CCA_Ax
  Description:      ConnectX-5 EN network interface card; 100GbE dual-port QSFP28; PCIe3.0 x16; tall bracket; ROHS R6
  PSID:             MT_0000000012
  PCI Device Name:  /dev/mst/mt4119_pciconf0
  Base GUID:        IDMASK
  Base MAC:         MACMASK
  Versions:         Current        Available
     FW             16.35.4030     N/A
     PXE            3.6.0902       N/A
     UEFI           14.29.0015     N/A

Shows active link on both servers

rdma link show
link mlx5_0/1 state ACTIVE physical_state LINK_UP netdev enp197s0f0np0
link mlx5_1/1 state ACTIVE physical_state LINK_UP netdev enp197s0f1np1

Also ibstat say ethernet link layer

CA 'mlx5_0'
        CA type: MT4119
        Number of ports: 1
        Firmware version: 16.35.4030
        Hardware version: 0
        Node GUID: 0x430492340923
        System image GUID: 0x423423423423
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xfd43431e598f
                Link layer: Ethernet
CA 'mlx5_1'
        CA type: MT4119
        Number of ports: 1
        Firmware version: 16.35.4030
        Hardware version: 0
        Node GUID: 0x430492340923
        System image GUID: 0x423423423423
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xfd43431e598f
                Link layer: Ethernet

ethtool -m enp197s0f0np0
        Identifier                                : 0x11 (QSFP28)
        Extended identifier                       : 0x00
        Extended identifier description           : 1.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Power set                                 : Off
        Power override                            : Off
        Connector                                 : 0x23 (No separable connector)
        Transceiver codes                         : 0x88 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Transceiver type                          : 40G Ethernet: 40G Base-CR4
        Transceiver type                          : 100G Ethernet: 100G Base-CR4 or 25G Base-CR CA-L
        Encoding                                  : 0x05 (64B/66B)
        BR, Nominal                               : 25500Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 0km
        Length (OM3 50um)                         : 0m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 3m
        Transmitter technology                    : 0xa0 (Copper cable unequalized)
        Attenuation at 2.5GHz                     : 6db
        Attenuation at 5.0GHz                     : 9db
        Attenuation at 7.0GHz                     : 12db
        Attenuation at 12.9GHz                    : 17db
        Vendor name                               : Mellanox
        Vendor OUI                                : 00:02:c9
        Vendor PN                                 : MCP1600-C003E30L
        Vendor rev                                : A6
        Vendor SN                                 : MTNUMBERSNREMOVED
        Date code                                 : 240306
        Revision Compliance                       : SFF-8636 Rev 2.0
        Module temperature                        : 0.00 degrees C / 32.00 degrees F
        Module voltage                            : 0.0000 V

So on ping or any data is transfered between nodes.

So now I am rely worried that my dac cables maybe not working any or I am missing something.

If some one has idea what I am doing wrong or how to check these cables would so grateful.

Thank you so much.

 

[waltar]

So on ping or any data is transfered between nodes.

You said it's ok ...

So now I am rely worried that my dac cables maybe not working any or I am missing something.

Or you are unshure ...

I'm quiet shure you need 3 subnets instead of 1 as the routing would not know which way to go, eg:
10.5.5.x + 10.5.6.x for node1
10.5.5.x + 10.5.7.x for node2
10.5.6.x + 10.5.7.x for node3

Link show 100Gbit active and so that's iO.

 

[tane]

I quite shure i got fried brain from compiling kernel drivers. Thank you so much I completely overlooked I am not using a switch . Thank you. Thank you Waltar . Sometime you need a just a small push from good people. thx man so much.
Anyway is is some is trying to work with lates promox and this card nvidia DOCA 2.8 drivers is working if you brian is not fried

 

[waltar]

If you work with zfs replication that may work with 3 subnets ... but I strongly assume your cluster corosync conf (need extra eg. 1G net now) don't like that nor a shared storage and maybe you will invest into a quiet costly switch also ...
PS: 100Gb OPA switch, the HFI cards and cables or even all as package are really cheap on used market place yet.