Mellanox Need help setting up Infiniband

Heini Skrypzeck

Active Member
Jun 16, 2019
2
0
41
43
Hi,

I have 2x ConnectX-2 Cards and need to setup infiniband, RDMA over NFS on Proxmox 5.4. I followed a few guides online but I'm unable to change protocol type from ETH to IB. What is correct way of setting the port type on Proxmox?
 
Hi,

I have 2x ConnectX-2 Cards and need to setup infiniband, RDMA over NFS on Proxmox 5.4. I followed a few guides online but I'm unable to change protocol type from ETH to IB. What is correct way of setting the port type on Proxmox?

The best way i found was by setting the options in the kernel module, and then rebuild the initfs.

```Found a workaround for now. Setting the port type manually in /etc/modprobe.d/mlx4.conf, then regenerating the initramfs image so it's read on boot works.

eg:

# echo options mlx4_core port_type_array=2 >> /etc/modprobe.d/mlx4.conf
# dracut --force
# grub2-mkconfig -o /boot/grub2/grub.cfg

The "port_type_array=2" option is the important bit. `modinfo mlx4_core` says
this about port_type_array:

port_type_array:Array of port types: HW_DEFAULT (0) is default 1 for IB, 2 for Ethernet (array of int)

So for my single port card, the single 2 value means "set it to ethernet mode".

With the grub2-mkconfig command, I'm not sure if it was needed. But it didn't seem to hurt things. :)

After this, the card is coming up fine in ethernet mode and working as per normal. :)```
https://bugs.centos.org/view.php?id=14419 and https://community.mellanox.com/s/article/howto-change-port-type-in-mellanox-connectx-3-adapter