Infiniband IPoIB routing problem

iPero

Member
Nov 11, 2018
3
0
6
23
Hi guys, i need some help with the IPoIB configuration, i've been trying to use infiniband to share storage between nodes unsuccessfully. I've installed rdma on both nodes and opensm on one node to manage the subnet, i'm using the default configuration which ,from what i can read online, seems to work most of the time. My cards are two MT25208 with firmware 5.2.0. What i get from ibstat is this:
Code:
CA 'mthca0'
        CA type: MT25208
        Number of ports: 2
        Firmware version: 5.2.0
        Hardware version: a0
        Node GUID: 0x0002c9020022f8a0
        System image GUID: 0x0002c9020022f8a3
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 2
                LMC: 0
                SM lid: 2
                Capability mask: 0x02510a6a
                Port GUID: 0x0002c9020022f8a1
                Link layer: InfiniBand
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 1
                LMC: 0
                SM lid: 1
                Capability mask: 0x02510a6a
                Port GUID: 0x0002c9020022f8a2
                Link layer: InfiniBand
which indicate that the cards are successfully connected and even running ibdiagnet shows no problems.
I have ib_ipoib module enabled on both machines and the interface comes up correctly, but if i asssign an ip address to both nodes and try to ping each other i get a 100% packet loss rate. Even ibping that seems to work in loopback doesn't work between the nodes. It's my first time configuring infiniband and i have 2 very old cards with not so much documentation online and i'm stuck. I would appreciate any help you could give me. I don't know which kind of info do you need but i'll add everything you nedd.
Thanks in advance,
Tommaso.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!