Mellanox Need help setting up Infiniband

Heini Skrypzeck

Active Member
Jun 16, 2019
2
0
41
43
Hi,

I have 2x ConnectX-2 Cards and need to setup infiniband, RDMA over NFS on Proxmox 5.4. I followed a few guides online but I'm unable to change protocol type from ETH to IB. What is correct way of setting the port type on Proxmox?
 
Hi,

I have 2x ConnectX-2 Cards and need to setup infiniband, RDMA over NFS on Proxmox 5.4. I followed a few guides online but I'm unable to change protocol type from ETH to IB. What is correct way of setting the port type on Proxmox?

The best way i found was by setting the options in the kernel module, and then rebuild the initfs.

```Found a workaround for now. Setting the port type manually in /etc/modprobe.d/mlx4.conf, then regenerating the initramfs image so it's read on boot works.

eg:

# echo options mlx4_core port_type_array=2 >> /etc/modprobe.d/mlx4.conf
# dracut --force
# grub2-mkconfig -o /boot/grub2/grub.cfg

The "port_type_array=2" option is the important bit. `modinfo mlx4_core` says
this about port_type_array:

port_type_array:Array of port types: HW_DEFAULT (0) is default 1 for IB, 2 for Ethernet (array of int)

So for my single port card, the single 2 value means "set it to ethernet mode".

With the grub2-mkconfig command, I'm not sure if it was needed. But it didn't seem to hurt things. :)

After this, the card is coming up fine in ethernet mode and working as per normal. :)```
https://bugs.centos.org/view.php?id=14419 and https://community.mellanox.com/s/article/howto-change-port-type-in-mellanox-connectx-3-adapter
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!