Looking for a "cheap" way to upgrade to 40gibt

[proxwolfe]

im getting about 40gbps on ipoib with the same hardware, you have a couple issues at play here.
1) mtu is not 4k
2) your lowest member is at 10gbps and not 40.

id get the output for:
- ibstat from each host
- ibnetdiscover
- ibdiagnet

and a screenshot if the ib switch sm settings, i do think the default for the subnet manager in ib switches is SDR which is 10gbps.

# ibstat
CA 'rocep101s0'
        CA type: MT4099
        Number of ports: 2
        Firmware version: 2.36.5000
        Hardware version: 1
        Node GUID: 0x248a07030078bb90
        System image GUID: 0x248a07030078bb93
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x268a07fffe78bb91
                Link layer: Ethernet
        Port 2:
                State: Active
                Physical state: LinkUp
                Rate: 40
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0x268a07fffe78bb92
                Link layer: Ethernet

# ibdiagnet
Loading IBDIAGNET from: /usr/lib/x86_64-linux-gnu/ibdiagnet1.5.7
-W- Topology file is not specified.
    Reports regarding cluster links will use direct routes.
Loading IBDM from: /usr/lib/x86_64-linux-gnu/ibdm1.5.7
-W- A few ports of local device are up.
    Since port-num was not specified (-p option), port 1 of device 1 will be
    used as the local port.
-E- Fail to ibvs_bind.

# ibnetdiscover
ibwarn: [169615] mad_rpc_open_port: client_register for mgmt 1 failed
./libibnetdisc/ibnetdisc.c:802; can't open MAD port ((null):0)
ibnetdiscover: iberror: failed: discover failed

the last two commands fail. Maybe because the cards are in eth mode now? Or should they work in eth mode as well?