Unable to see Infiniband interfaces in Proxmox?

Re: Unable to see Infiniband interfaces on Proxmox?

What are the performance difference between eth-mode and ib-mode?
Do you use a special cable with SFF-8470 to RJ-45?
Can you connect directly to an ethernet switch?
 
Re: Unable to see Infiniband interfaces on Proxmox?

mir said:
What are the performance difference between eth-mode and ib-mode?
Do you use a special cable with SFF-8470 to RJ-45?
Can you connect directly to an ethernet switch?
Click to expand...

For 10gigE vs SDR IB it's 10Gbit/s vs 8Gbit/s. At least in theory.

I do not know what the actual performance difference is, but in my case IPoIB theoretical limit is higher at 16Git/sec as I have a DDR IB card. But I have no IB switch.

However I will be plugging it into a network switch with 4x 10gigE SFP+ uplinks.
Once it's in native 10gigE mode there should be no reason for it not to work.

I have a CX4 to SFP+ cable coming in the mail.

Another option is CX4-to-QSFP cable in conjunction with a QSFP-to-SFP+ adapter but tha's way more expensive.
 
Last edited:
Re: Unable to see Infiniband interfaces on Proxmox?

swartz said:
CX4 to SFP+ cable coming in the mail.

Another option is CX4-to-QSFP cable in conjunction with a QSFP-to-SFP+ adapter but tha's way more expensive.
Click to expand...

I've never heard about such cables. Could you point me out to such one? Did you manage to get it working?
From my understanding CX4 to SPF+ could be achieved only with active media converter due to the differences in physical nature of both cables.
 
Re: Unable to see Infiniband interfaces on Proxmox?

Whatever said:
I've never heard about such cables. Could you point me out to such one? Did you manage to get it working?
From my understanding CX4 to SPF+ could be achieved only with active media converter due to the differences in physical nature of both cables.
Click to expand...


I haven't got mine yet. Still in the mail.
From my research, no active media converter is needed for this type of cable.

I got mine from ebay, but I can't find any one. Once in a while they pop up.

I don't know where you are, but here is another source:
www. datastoragecables .com/cx4/CX4-SFP+/
 
Re: Unable to see Infiniband interfaces on Proxmox?

I've got it. It's an Ethernet cable but with CX-4 and SFP+ connectors (10GBASE-CX4)
In my environment I'm using (like mir) an infiniband switch (SFS 7000 / Topspin 120) and couple of IB cards on 25408 chipset. Seems my cables (which are used to connect IB switch and IB card like https://www.datastoragecables.com/infiniband/4x-4x/eject-eject/C5252-ME-DDR.html) will not work if I connect IB card (it's VPI so Ethernet 10Gb should be supported) to 10GBASE card like Intel 10gbe 10 Gigabit PCI-E dual-port 10gbase-cx4 (expx9502cx4)

Am I correct?
 
Re: Unable to see Infiniband interfaces on Proxmox?

Whatever said:
I've got it. It's an Ethernet cable but with CX-4 and SFP+ connectors (10GBASE-CX4)
Click to expand...
Yes, it's just a passive cable with CX4 and SFP+ on opposite ends.

Whatever said:
Seems my cables (which are used to connect IB switch and IB card like https://www.datastoragecables.com/infiniband/4x-4x/eject-eject/C5252-ME-DDR.html) will not work if I connect IB card (it's VPI so Ethernet 10Gb should be supported) to 10GBASE card like Intel 10gbe 10 Gigabit PCI-E dual-port 10gbase-cx4 (expx9502cx4)

Am I correct?
Click to expand...

Your VPI card has to be switched into native 10gig Ethernet mode. Not IPoIB.
That means special changes need to be made to load the right modules. Read my posts in this thread.

Once the VPI card is running in native 10gigE mode, you should be able to connect to your intel card.

PS. Also make sure you got the latest firmware for your VPI card.
 
Re: Unable to see Infiniband interfaces on Proxmox?

So I finally got the CX4 -> SFP+ cable. The cards is set to native 10gigE mode. Connected an ethernet switch 10gig uplink and... getting extremely slow speed.

To troubleshoot I have quickly thrown together 2 Proxmox test systems ConnectX cards, same firmware, The HCAs are connected directly with CX4 cable.
With IPoIB in I am getting 6-7Gb/s in iperf. When I switch the cards to 10gigE mode, iperf seems to hang indefinitely and whenever it finishes the measured link speed is so slow that it's listed in kbit/sec.

Any ideas what might be wrong?
 
Re: Unable to see Infiniband interfaces on Proxmox?

Hi,

i'm using connect-x3 card here (ethernet mode), with mellanox sx1012 ethernet switchs, without any problem.

When I switch the cards to 10gigE mode, iperf seems to hang indefinitely and whenever it finishes the measured link speed is so slow that it's listed in kbit/sec.
Click to expand...


are your switch ethernet or infiniband ?


 
Re: Unable to see Infiniband interfaces on Proxmox?

spirit said:
Hi,

i'm using connect-x3 card here (ethernet mode), with mellanox sx1012 ethernet switchs, without any problem.



are your switch ethernet or infiniband ?


[/COLOR]
Click to expand...

I initially plugged them into an ethernet switch (gigE switch with 4 10gigE uplink ports). To eliminate switch or the CX4->SFP+ cable as the cause of the problem, I took the two systems with same ConnectX-1 DDR cards and connected the two cards directly with a passive CX4 cable.

With this direct card-to-card connection and tested via iperf I get these results.
IPoIB:
Around 6-7Gbit/s with datagram mode and default 2044 MTU.
In connected mode with 65520 MTU I get 1-2Gbit/sec. Strange but not interested to investigate as I want native 10gigE to work.

Ethernet:
With native 10gigE mode I'm lucky to get 200-300 kbit/sec.

Firmware is latest available for these card v2.9.1000.
 
Last edited:
Re: Unable to see Infiniband interfaces on Proxmox?

I fanally got my Intel 10GB ethernet card (EXPX9502CX4) and connected it to MT25418 [ConnectX VPI] switched to Ethernet mode with passive CX4 cable.

Interfaces seem up but no link detected

Code: 
[root@localhost ~]# ethtool enp3s0
Settings for enp3s0:
        Supported ports: [ TP ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no

Should the copper be somehow ethernet specific or something goes wrong with Intel 10GBe card?

Any ideas are very appreciated
 
Re: Unable to see Infiniband interfaces on Proxmox?

Only downgrade MT25418 to 2.7.0 firmware did the trick. Very disappointed :(
 
Last edited:
Re: Unable to see Infiniband interfaces on Proxmox?

MT25418 [ConnectX VPI] (Proxmox Kernel 3.10-10) <==> EXPX9502CX4 (FreeBSD 9.1)

Iperf results:

Code: 
root@pve01A:~# iperf -c 10.10.10.254
------------------------------------------------------------
Client connecting to 10.10.10.254, TCP port 5001
TCP window size: 95.8 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.1 port 55042 connected with 10.10.10.254 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  6.31 GBytes  5.42 Gbits/sec


root@pve01A:~# iperf -c 10.10.10.254 -u
------------------------------------------------------------
Client connecting to 10.10.10.254, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size:  208 KByte (default)
------------------------------------------------------------
[  3] local 10.10.10.1 port 45377 connected with 10.10.10.254 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec
[  3] Sent 893 datagrams
[  3] Server Report:
[  3]  0.0-10.0 sec  1.25 MBytes  1.05 Mbits/sec   0.760 ms    0/  893 (0%)

Changing mtu to 9000 makes no difference
 
Re: Unable to see Infiniband interfaces on Proxmox?

swartz said:
Firmware is latest available for these card v2.9.1000.
Click to expand...

I found couple of non-released firmware sources for MT25408: 2.9.1100, 2.9.1200, 2.9.1290.
You can try to make a firmware and burn it to your card to check if it helps.

If you want to try - drop me a line
 
Re: Unable to see Infiniband interfaces on Proxmox?

Whatever said:
I found couple of non-released firmware sources for MT25408: 2.9.1100, 2.9.1200, 2.9.1290.
You can try to make a firmware and burn it to your card to check if it helps.

If you want to try - drop me a line
Click to expand...


Yes, I'd be very much interested in obtaining these later firmwares.
Will send you a PM just in case.

Edit: turns out there is no PM enabled here.
 
Last edited:
Re: Unable to see Infiniband interfaces on Proxmox?

contact me by email: matveev.as (at) gmail.com
 
Re: Unable to see Infiniband interfaces on Proxmox?

I was attempting to set up IB networking on Proxmox 4 and ran into the same problems as others on this thread are reporting. The mlx4_core module loads by default and the hba is initialized at boot, but following the guide at http://pkg-ofed.alioth.debian.org/ didnt result on initialized interfaces.

The solution, at least for my elderly ConnectX2 VPI cards, is to include mlx4_ib to /etc/modules-load.d/modules.conf IN ADDITION to the other modules you wish to use, eg ib_ipoib (you dont need to specify mlx4_core.) This may or may not be required for newer cards but I figured I'll leave this here for anyone running into the same problem.
 
I was never able to get 1st gen ConnectX cards to work in native 10gigE mode. Even with these later firmware (provided by Whatever). I suspect mlx4_en is broken or dropped support for 1st gen cards now that they are EOL-ed.

alexskysilk,
Please note, when you run cards in Infiniband mode (e.g. IPoIB) you need to run a subnet manager (SM). If you don't have a switch with SM or when you connect HCAs directly, then one of the nodes must run SM. Easiest way is to:

apt-get install opensm
service opensm
 
I do not have a switch with SM so I have installed opensm on every proxmox host. The god thing about SM is that it has built in master/slave functionality so as soon as the current master disappear the remaining SM on the other hosts will renegotiate for a new master.
 
swartz said:
I was never able to get 1st gen ConnectX cards to work in native 10gigE mode. Even with these later firmware (provided by Whatever). I suspect mlx4_en is broken or dropped support for 1st gen cards now that they are EOL-ed.

alexskysilk,
Please note, when you run cards in Infiniband mode (e.g. IPoIB) you need to run a subnet manager (SM). If you don't have a switch with SM or when you connect HCAs directly, then one of the nodes must run SM. Easiest way is to:

apt-get install opensm
service opensm
Click to expand...

True- and I completely glossed over it as I'm always using SM capable switches. Thats a pretty important point :)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom