What NICs 10Gb are compatible with Proxmox VE? and related matters

[e100]

Hello e100,

How did you get Proxmox to recognize the Mellanox MHEA28-XTC? I have installed these and for none of the servers could recognize it. It does not show up in the Proxmox GUI or IFCONFIG.

Any help would be greatly appreciated. Thank you.

IP is not native to infiniband, you need to load the ipoib kernel module. Once loaded you should have a ib0 and ib1 interface show up in ifconfig.

I use a pre-up (I think) statement in /etc/network/interfaces to run "modprobe ipoib" before starting the IB interfaces.

I hope this is enough to point you in the right direction, I'm on vacation and do not have simple access to look at my configs to give more specific details.

Also, you need an Infiniband "subnet manager" running somewhere on your IB network. On a node using opensm or in your switch if supported.

 

[udo]

Hello e100,

How did you get Proxmox to recognize the Mellanox MHEA28-XTC? I have installed these and for none of the servers could recognize it. It does not show up in the Proxmox GUI or IFCONFIG.

Any help would be greatly appreciated. Thank you.

Hi,
for infiniband-cards, like e100 allready wrote, is further configuration necessary. Look here: http://pve.proxmox.com/wiki/Infiniband

For my new card I must load more modules:

auto ib0
iface ib0 inet static
        address  172.20.9.64
        netmask  255.255.255.0
        pre-up modprobe mlx4_ib ; modprobe ib_ipoib; modprobe ib_umad
        pre-up echo connected > /sys/class/net/ib0/mode
        mtu 65520

Udo

 

[impire]

IP is not native to infiniband, you need to load the ipoib kernel module. Once loaded you should have a ib0 and ib1 interface show up in ifconfig.

I use a pre-up (I think) statement in /etc/network/interfaces to run "modprobe ipoib" before starting the IB interfaces.

I hope this is enough to point you in the right direction, I'm on vacation and do not have simple access to look at my configs to give more specific details.

Also, you need an Infiniband "subnet manager" running somewhere on your IB network. On a node using opensm or in your switch if supported.

Thank you kindly for taking the precious time out of your vacation to reply. That is very nice of you. I am a newbie so pardon the next question2.

1) What is the command to install the ipoib kernel module?

Not sure if this is the correct step-by-step documentation on setting up Infiniband, but I have tried to follow this looks different and doesn't mention about the ipoib kernet:

http://pkg-ofed.alioth.debian.org/howto/infiniband-howto.html

2) What switch are you using? I have a choice now between a Cisco SFS7000D and Flextronics 9024D switch (both 20Gbps). Does it matter which switch I use? I worry about compatibility between the network cards and the switches. Flextronicx uses Mellanox firmware.

 

[impire]

Hi,
for infiniband-cards, like e100 allready wrote, is further configuration necessary. Look here: http://pve.proxmox.com/wiki/Infiniband

For my new card I must load more modules:

auto ib0
iface ib0 inet static
        address  172.20.9.64
        netmask  255.255.255.0
        pre-up modprobe mlx4_ib ; modprobe ib_ipoib; modprobe ib_umad
        pre-up echo connected > /sys/class/net/ib0/mode
        mtu 65520

Udo

Thank you very much. Is this considered the official guide for step-by-step install/configure of Infiniband on ProxMox? The wiki did not mentioned about loading the kernel. It would be nice if it did. As of now, I do not know how to load the correct kernel so it can recognize the NIC cards.

 

[udo]

Thank you very much. Is this considered the official guide for step-by-step install/configure of Infiniband on ProxMox?

Hi,
it's an wiki - any registered user can update the content.

The wiki did not mentioned about loading the kernel. It would be nice if it did. As of now, I do not know how to load the correct kernel so it can recognize the NIC cards.

There is no special kernel needet - all things work with the pve-kernel (for all IB-cards I use).

Udo

 

[impire]

Hi,

There is no special kernel needet - all things work with the pve-kernel (for all IB-cards I use).

Udo

Thanks.

- The reason I post to this thread (and to e100) was because none of the Mellanox cards that I've installed are recognized by ProxMox. I am using version 3.2. I have tried the following NIC cards:

MHGS18-XTC, MHET2X-1TC, MHEA28-XTC (e100 stated it works).

- e100 mentioned he had to load "the ipoib kernel module". But you mentioned no special kernel needed. I am now confused.

- I am even further confused when this link on debian.org mentioned to load the Infiniband kernel modules:

http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-4.html

Could you please clarify the step-by-step guide needed in order to get Infiniband to work with Proxmox? Thank you very much.

 

[udo]

Thanks.

- The reason I post to this thread (and to e100) was because none of the Mellanox cards that I've installed are recognized by ProxMox. I am using version 3.2. I have tried the following NIC cards:

MHGS18-XTC, MHET2X-1TC, MHEA28-XTC (e100 stated it works).

- e100 mentioned he had to load "the ipoib kernel module". But you mentioned no special kernel needed. I am now confused.

Hi,
kernel module != kernel... you ask for an special kernel - answer no! But of yourse you need perhaps the right kernel module.

And like I wrote before: I must load the modules: mlx4_ib ib_ipoib ib_umad
(this is done with the ib0/pre-up section in the interface-file).

Udo

 

[impire]

Hi,
kernel module != kernel... you ask for an special kernel - answer no! But of yourse you need perhaps the right kernel module.

And like I wrote before: I must load the modules: mlx4_ib ib_ipoib ib_umad
(this is done with the ib0/pre-up section in the interface-file).

Udo

Thank you. I did as suggested. Now I see following columns in the Network section.

Name ib0-ib4
Type = Unknown
Active = No
Autostart = Yes

Is this normal? eth0 and eth1 is showing Type = Network. But the ib0-ib4 is showing as Unknown".

 

[impire]

Thank you. I did as suggested. Now I see following columns in the Network section.

Name ib0-ib4
Type = Unknown
Active = No
Autostart = Yes

Is this normal? eth0 and eth1 is showing Type = Network. But the ib0-ib4 is showing as Unknown".

Hello,

Can anyone help? Is this normal to see the Mellanox Infiniband Network cards showing up as "Unknown" device?

 

[impire]

Hello,

Can anyone help? Is this normal to see the Mellanox Infiniband Network cards showing up as "Unknown" device?

Deafening silence. I guess trying to get infiniband to work with Proxmox is more complex than it seems. I can't even get it to pass the initial stage of recognizing the network card. This is definitely not plug and play, but rather a plug and pray.

 

[udo]

Deafening silence. I guess trying to get infiniband to work with Proxmox is more complex than it seems. I can't even get it to pass the initial stage of recognizing the network card. This is definitely not plug and play, but rather a plug and pray.

Hi,
it's quite normal, because the gui is to handle interfaces for bridges (to connect VMs). Bridges has an MTU of 1500 - and you don't want to use infiniband with this MTU-size!

E.G. you configure your ib-interface in the interface-file and use it for drbd or storage-network...

See screenshot

Udo

 

[jaysol]

Chelsio also work out of the box. Iperf from chelsio to an solarflare gives 5.63 Gbits/sec (and 5.41 the other way).

Udo

**WARNING** to future low budget 10GBE NIC purchasers.... Chelsio T420-CR does not work OOB here with PVE 6.x. If you're finding these old threads, be sure to research well before you buy.

This was incorrect. The Chelsio T420-CR (and likely T5xx and T6xx from same driver series) did work OOB / out of box with PVE. If I remember correctly, the interfaces showed up as ens1 > with ens1f4 ens1f4d1 being the two ports.

Therefore, The Chelsio T4, T5, T6 cards make a very cost effective 10G solution. I can not speak towards performance yet though.