[SOLVED] pve 6.3 vs. mellanox ofed

[reckless]

Still not clear to me if you CAN enable RoCev2 in the default drivers or if that's impossible.

 

[dan.ger]

As I told you I‘m fine with the network performance and latency. So I did not try to enable RoCev2 with default kernel driver.

 

[78Star]

So I checked for buildin drivers, RoCev2 is not enabled, but the default driver performs for my setup better than the original Mellanox drivers. Latency is below 0.035ms and throughput roudabout 96GBps. I think that is ok. I have no issues with the virtual machines and ceph, no freezing, no delay, no slow ops.

And don't forget to run your cpu at maximum performance without powersaving, c-state, speedstep and so on to geht maximum performance of the mellanox cards.

Can you source a link for how to enable source drivers, and if it works for connectx-3?

 

[dan.ger]

So if you do not have installed proxmox 7.x and you have a working cluster, then remove a node from the cluster (Cluster Manager Tutorial) and install proxmox 7.x. Proxmox 7.x will provide the default driver for mellanoc (mlnx4/mlnx5), so you do not have to install additional things. After you install the new node, join it to existing cluster, all configurations are provided by the cluster to the new node, but you have to name it to same name as before. Go on with all other nodes.

If you did an inplace upgrade you have to deinstall old mellanox drivers (apt purge mlnx-ofed-basic), reboot the node, fix the network interface names and you are done, Do this node by node.

If you are installing new nodes and cluster, just install Proxmox 7.x and create your cluster.

 

[klpzk]

Tried this with PVE 7.x fresh install and Connect-X3. The drivers are there, and loaded automatically, however the interface does not show up in ip a results nor any other IP management tools. It is still unclear if these default drivers were set up to work with IP networks or IB networks. The setup and basic usage to achieve the results discussed in this thread would benefit some.

 

[dan.ger]

Are you sure that the mode is set to Ethernet for your card?

 

[klpzk]

As @78Star mentioned, is there a link, documentation you are using enable these features with the default drivers? I don't see a way to mirror your setup without using an up-to-date version of mlnx-ofed.