Simple setup :
1 Dell t630 with:
----------------------------------------------------------------
lspci |grep Mell
02:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev a0)
------------------------------------------------------------------------------
2. Windows 10 machine with the same Mellanox card.
Both machines connected directly with qsfp+ AOC cabel.
On proxmox side:
-------------------------------------------------------------------------------------------
ibstatus
Infiniband device 'ibp2s0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:000b:9bed
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
Infiniband device 'ibp2s0' port 2 status:
default gid: fe80:0000:0000:0000:0002:c903:000b:9bee
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
-----------------------------------------------------------------------------------------------------------------------
On proxmox gui site visible ibp2s0 with 192.168.40.221/24. When starting opensm on proxmox :
oot@proxmox:~# opensm
-------------------------------------------------
OpenSM 3.3.23
Reading Cached Option File: /etc/opensm/opensm.conf
Loading Cached Option:sm_key = 0x0000000000000000
Command Line Arguments:
Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 3.3.23
Using default GUID 0x2c903000b9bed
Entering DISCOVERING state
Entering MASTER state
-------------------------------------------
On windows site Mellanox has assigned 192.168.40.22/24, after opensm.exe I get :Entering STANBY state. If i run opensm on windows first then get :
--------------------------------------
Entering MASTER State
SUBNET UP
---------------------------------------
Cant ping between windows and proxmox ? connection can't be established. Need expert advise
.................
1 Dell t630 with:
----------------------------------------------------------------
lspci |grep Mell
02:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev a0)
------------------------------------------------------------------------------
2. Windows 10 machine with the same Mellanox card.
Both machines connected directly with qsfp+ AOC cabel.
On proxmox side:
-------------------------------------------------------------------------------------------
ibstatus
Infiniband device 'ibp2s0' port 1 status:
default gid: fe80:0000:0000:0000:0002:c903:000b:9bed
base lid: 0x1
sm lid: 0x1
state: 4: ACTIVE
phys state: 5: LinkUp
rate: 40 Gb/sec (4X QDR)
link_layer: InfiniBand
Infiniband device 'ibp2s0' port 2 status:
default gid: fe80:0000:0000:0000:0002:c903:000b:9bee
base lid: 0x0
sm lid: 0x0
state: 1: DOWN
phys state: 2: Polling
rate: 10 Gb/sec (4X SDR)
link_layer: InfiniBand
-----------------------------------------------------------------------------------------------------------------------
On proxmox gui site visible ibp2s0 with 192.168.40.221/24. When starting opensm on proxmox :
oot@proxmox:~# opensm
-------------------------------------------------
OpenSM 3.3.23
Reading Cached Option File: /etc/opensm/opensm.conf
Loading Cached Option:sm_key = 0x0000000000000000
Command Line Arguments:
Log File: /var/log/opensm.log
-------------------------------------------------
OpenSM 3.3.23
Using default GUID 0x2c903000b9bed
Entering DISCOVERING state
Entering MASTER state
-------------------------------------------
On windows site Mellanox has assigned 192.168.40.22/24, after opensm.exe I get :Entering STANBY state. If i run opensm on windows first then get :
--------------------------------------
Entering MASTER State
SUBNET UP
---------------------------------------
Cant ping between windows and proxmox ? connection can't be established. Need expert advise
.................