Infiniband 40gbps capped at 12.5gbps

mir said:
You could try temporarily to "down-grade" in this file: /etc/debian_version
Click to expand...
I changed the debian_version to 7.5 and installation of driver seems to start, however it aborts with following error:
Error: One or more packages depends on MLNX_OFED.
Those packages should be removed before uninstalling MLNX_OFED:

glusterfs-common spiceterm pve-qmeu-kvm libpve-storage-perl qemu-server pve-manager
Click to expand...
I am pretty sure it would be very bad thing for the proxmox node if i do remove them.
 
Is this what you see when installing a binary package or the source package?

You could try installing using: dpkg --force-depends -i package.deb

This should install the package disregarding the dependencies. As the error states the package will not remove the dependent packages itself.
 
mir said:
Is this what you see when installing a binary package or the source package?

You could try installing using: dpkg --force-depends -i package.deb

This should install the package disregarding the dependencies. As the error states the package will not remove the dependent packages itself.
Click to expand...

The driver package came with its own installer script. I am only running the script. The driver folder have .deb packages in DEBS subfolder along documentation. The drive package was downloaded from here: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers, for Debian 7.5 in .tgz. Should i try to run all the debs packages individually? But i am assuming if i do that it will asks for dependencies for every one of the debs in the sub folder.
 
You could also hope for a quick update to RHEL 6.6 kernel in proxmox.
https://access.redhat.com/documenta...ml/6.6_Technical_Notes/ch-device_drivers.html


  • All Mellanox mlx drivers have been updated to their latest upstream versions.
  • The ixbe driver has been updated to the latest upstream version.

And also:
[h=2]4.2. Networking[/h]


Mellanox SR-IOV SupportSingle Root I/O Virtualization (SR-IOV) is now supported as a Technology Preview in the Mellanox libmlx4 library and the following drivers:

  • mlx_core
  • mlx4_ib (InfiniBand protocol)
  • mlx_en (Ethernet protocol)

Package: kernel-2.6.32-335
 
Last edited:
mir said:
Maybe: dpkg --force-depends -i DEBS/*.deb
Click to expand...
I tried this command to install OFED drivers. Although the installation seem to go ok, after reboot none of the infiniband drivers were loaded. The installation appears to remove mlx4_ib drivers and other protocols. So based on the manual i used the uninstaller came with the package to uninstall OFED drivers only find out that it also removed packages such as pve, qemu and soem other. Unfortunately it did not ask me do i want to remove or cancel. It only showed what has been removed. So after a reboot i found out that services such as pveproxy, pvedaemon, pvestatd are missing and thus the node shows offline from GUI. This is a ceph node so i dont necessarily need qemu packages. The ceph OSDs are working just fine though.

Which packages do i need to install ir order get those services back. I am able to see content of /etc/pve/ so looks like cluster side is working.
 
I have just installed Mellanox ConnectX DDR and first test run with iperf:
iperf -s -m -B 10.0.0.1
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.0.1
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.0.0.1 port 5001 connected with 10.0.0.1 port 42385
bind failed: Address already in use
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
Binding to local address 10.0.0.1
TCP window size: 530 KByte (default)
------------------------------------------------------------
[ 6] local 10.0.0.1 port 42386 connected with 10.0.0.1 port 5001
[ 5] local 10.0.0.1 port 5001 connected with 10.0.0.1 port 42386
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-30.0 sec 47.2 GBytes 13.5 Gbits/sec
[ 4] MSS size 16384 bytes (MTU 16424 bytes, unknown interface)
[ 6] 0.0-30.0 sec 58.6 GBytes 16.8 Gbits/sec
[ 6] MSS size 16384 bytes (MTU 16424 bytes, unknown interface)
[ 5] 0.0-30.0 sec 58.6 GBytes 16.8 Gbits/sec
[ 5] MSS size 16384 bytes (MTU 16424 bytes, unknown interface)
[SUM] 0.0-30.0 sec 106 GBytes 30.3 Gbits/sec


iperf -c 10.0.0.1 -B 10.0.1.1 -i5 -t30 -d -m
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.1.1
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
bind failed: Address already in use
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
Binding to local address 10.0.1.1
TCP window size: 331 KByte (default)
------------------------------------------------------------
[ 5] local 10.0.0.1 port 42385 connected with 10.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 5] 0.0- 5.0 sec 8.09 GBytes 13.9 Gbits/sec
[ 5] 5.0-10.0 sec 7.57 GBytes 13.0 Gbits/sec
[ 5] 10.0-15.0 sec 8.15 GBytes 14.0 Gbits/sec
[ 5] 15.0-20.0 sec 8.02 GBytes 13.8 Gbits/sec
[ 5] 20.0-25.0 sec 8.17 GBytes 14.0 Gbits/sec
[ 5] 25.0-30.0 sec 7.20 GBytes 12.4 Gbits/sec
[ 5] 0.0-30.0 sec 47.2 GBytes 13.5 Gbits/sec
[ 5] MSS size 16384 bytes (MTU 16424 bytes, unknown interface)

Remember above is performance per connection and since this is running a dual test the numbers is showed per connection.

I am still waiting for my cables but I cannot wait until they arrive ;-)
 
Hi symmcom,

Did you remember this trick provided you have dual port HCA's?
cat /etc/sysctl.d/infiniband.conf
net.ipv4.conf.ib0.arp_ignore=1
net.ipv4.conf.ib1.arp_ignore=1
 
If your DDR IB card putting out that much bandwidth, my QDR cards must put out at least closer to yours. Below is a snippet of output after running your exact command. Not good.
Code: 
root@pmx-ceph07:~# iperf -c 10.0.100.17 -B 10.0.100.20 -i5 -t30 -d -m
------------------------------------------------------------
Server listening on TCP port 5001
Binding to local address 10.0.100.20
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
bind failed: Address already in use
------------------------------------------------------------
Client connecting to 10.0.100.17, TCP port 5001
Binding to local address 10.0.100.20
TCP window size: 2.27 MByte (default)
------------------------------------------------------------
[  5] local 10.0.100.20 port 39061 connected with 10.0.100.17 port 5001
[  4] local 10.0.100.20 port 5001 connected with 10.0.100.17 port 45729
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 5.0 sec  10.5 GBytes  18.1 Gbits/sec
[  5]  0.0- 5.0 sec  2.52 GBytes  4.33 Gbits/sec
[  4]  5.0-10.0 sec  10.4 GBytes  17.9 Gbits/sec
[  5]  5.0-10.0 sec  2.38 GBytes  4.09 Gbits/sec
[  5] 10.0-15.0 sec  3.09 GBytes  5.32 Gbits/sec
[  4] 10.0-15.0 sec  10.2 GBytes  17.6 Gbits/sec
[  5] 15.0-20.0 sec  4.46 GBytes  7.66 Gbits/sec
[  4] 15.0-20.0 sec  9.23 GBytes  15.9 Gbits/sec
[  5] 20.0-25.0 sec  4.71 GBytes  8.10 Gbits/sec
[  4] 20.0-25.0 sec  9.01 GBytes  15.5 Gbits/sec
[  5] 25.0-30.0 sec  5.61 GBytes  9.65 Gbits/sec
[  5]  0.0-30.0 sec  22.8 GBytes  6.52 Gbits/sec
[  5] MSS size 65480 bytes (MTU 65520 bytes, unknown interface)
[  4] 25.0-30.0 sec  8.28 GBytes  14.2 Gbits/sec
[  4]  0.0-30.0 sec  57.7 GBytes  16.5 Gbits/sec
[  4] MSS size 32768 bytes (MTU 32808 bytes, unknown interface)

Which driver did you use? Is it on Proxmox node?
 
It is installed in the Proxmox node.
From /etc/modules
# infiniband
# Mellanox ConnectX cards
mlx4_ib
#ib_mthca # some mellanox cards
#iw_cxgb3 # Chelsio T3 cards
#iw_nes # NetEffect cards


# Protocol modules
# Common modules
rdma_ucm
ib_umad
ib_uverbs
# IP over IB
ib_ipoib
# scsi over IB
ib_srp

mlx4_ib is the default one for kernel-2.6-32
modinfo mlx4_ib
filename: /lib/modules/2.6.32-32-pve/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko
version: 1.0
license: Dual BSD/GPL
description: Mellanox ConnectX HCA InfiniBand driver
author: Roland Dreier
srcversion: B9ACC7DFDC1537618F2F383
depends: mlx4_core,ib_core,ib_mad,ib_sa
vermagic: 2.6.32-32-pve SMP mod_unload modversions
parm: sm_guid_assign:Enable SM alias_GUID assignment if sm_guid_assign > 0 (Default: 1) (int)
Code: 
01:00.0 InfiniBand: Mellanox Technologies MT25408 [ConnectX VPI - IB SDR / 10GigE] (rev a0)
	Subsystem: Mellanox Technologies Device 0003
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 28
	Region 0: Memory at fe800000 (64-bit, non-prefetchable) [size=1M]
	Region 2: Memory at fb800000 (64-bit, prefetchable) [size=8M]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] Vital Product Data
		Product Name: Eagle IB DDR
		Read-only fields:
			[PN] Part number: HCA-00001            
			[EC] Engineering changes: A5
			[SN] Serial number: ML4209003684            
			[V0] Vendor specific: HCA 500Ex-D     
			[RV] Reserved: checksum good, 0 byte(s) reserved
		Read/write fields:
			[V1] Vendor specific: N/A    
			[YA] Asset tag: N/A                             
			[RW] Read-write area: 107 byte(s) free
		End
	Capabilities: [9c] MSI-X: Enable+ Count=256 Masked-
		Vector table: BAR=0 offset=0007c000
		PBA: BAR=0 offset=0007d000
	Capabilities: [60] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Kernel driver in use: mlx4_core



From /etc/network/interfaces
auto ib0
iface ib0 inet static
address 10.0.0.1
netmask 255.255.255.0
pre-up echo connected > /sys/class/net/ib0/mode
mtu 65520


auto ib1
iface ib1 inet static
address 10.0.1.1
netmask 255.255.255.0
pre-up echo connected > /sys/class/net/ib1/mode
mtu 65520
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom