infiniband configuration questions

e100 · Aug 7, 2013

The snmp server is inside of the switch, you do not need to setup anything on linux.
You configure the snmp server in the switch and the client application connects to it.

RobFantini · Aug 7, 2013

e100 said:
The snmp server is inside of the switch, you do not need to setup anything on linux.
You configure the snmp server in the switch and the client application connects to it.

I'm at the point that I can log in to the switch. That means I assume that snmp user and password are OK ?

Now I get ' Could not open trap port: Permission denied'
Any suggestions to fix that? here is my snmp relatred config:

Code:

username admin community-string  passwordxx
snmp-server user admin enable         
snmp-server user admin privilege unrestricted-rw
snmp-server enable traps authentication

e100 · Aug 7, 2013

I never bothered to setup traps, I just ignore that message as traps are not needed.

http://en.wikipedia.org/wiki/Simple_Network_Management_Protocol#Trap

RobFantini · Aug 7, 2013

Subnet Manager . [sm]

I've a cable attaching both systems. However they do not seem to see each other.

Does a port used to conect both systems need to be configured? Or is a certain port number suppposed to be used?

I have been reading manuals and searching....TopspinOS_RedundancyConfigGuide.pdf from the iso does not refer to model 120 . Is there another manual hwich explains sm?

e100 · Aug 8, 2013

You can configure different subnets on each switch or the same subnet on both. You want the same on both.

Log into the CLI on the switch and run

Code:

Show config

Does your config look like mine?
My config is pretty much bare minimum, I do not recall if having time synced is needed or not. I always setup NTP on all my devices so logs have proper time stamps.

I can plug any port on one into any port on another and it works just fine.

Do the lights on the ports lite up when you connect the cable? I believe there is a green and orange lite. If no lights maybe a bad cable?

The topology view in EM will show all links including links to servers.

The topspin 120 and Cisco sfs 7000 are the same thing that might help you decipher themanuals better.

RobFantini · Aug 8, 2013

the lites are green, here is our config.

Code:

!   TopspinOS-2.9.0/build170
!   Thu Aug  8 13:16:16 2013
enable
terminal time-out 60
config terminal
!
boot-config primary-image-source TopspinOS-2.9.0/build170
!
ib sm subnet-prefix fe:80:00:00:00:00:00:00 priority 0
!
interface mgmt-ethernet
 ip address 10.1.10.32 255.255.255.0
 gateway 10.1.10.1
 no shutdown
!
location "north pole"
!
ip domain-name fantinibakery.com
!
ntp server-one 10.1.0.2
ntp server-two 10.1.10.40
!
logging-server one 10.1.10.20

username admin password 7 xxxxxxxxx   
username admin community-string  xxxxxxxx
!
snmp-server user admin enable
snmp-server user admin privilege unrestricted-rw
!
snmp-server contact "admin"
hostname "Topspin-32"
snmp-server enable traps authentication

I'll check the Cisco sfs 7000 manuals..

e100 · Aug 8, 2013

I do not see anything wrong with your config.
You have both configured the same except for the IP address?

What is the output of this on both switches?:

Code:

show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00

I just looked on my switches, only a green light, blinks when there is traffic.

RobFantini · Aug 8, 2013

just the IP address is different.

Code:

Topspin-33# show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00


================================================================================
                      Discovered Subnet Managers in Fabric
================================================================================
            subnet-prefix : fe:80:00:00:00:00:00:00
                port-guid : 00:02:c9:02:00:27:26:b9
                 priority : 0
                 sm-state : master
                   sm-key : 00:00:00:00:00:00:00:00
                act-count : 643714

Code:

Topspin-32# show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00


================================================================================
                      Discovered Subnet Managers in Fabric
================================================================================
            subnet-prefix : fe:80:00:00:00:00:00:00
                port-guid : 00:02:c9:02:00:27:26:b9
                 priority : 0
                 sm-state : master
                   sm-key : 00:00:00:00:00:00:00:00
                act-count : 643700

e100 · Aug 8, 2013

The two switches are not seeing each other, not sure why tho.

Do this on each switch, change # to the port your cable is connected to:

Code:

show interface ib #

e100 · Aug 8, 2013

Is the port-guid really the same on both?

Code:

port-guid : 00:02:c9:02:00:27:26:b9

RobFantini · Aug 8, 2013

Code:

Topspin-32# show interface ib 1


================================================================================
                        InfiniBand Interface Information
================================================================================
                     port : 1
                     name : 1
                     type : ib4xTX
                     desc : 1 (65)
              last-change : none
                      mtu : 2048
 auto-negotiate-supported : no
           auto-negotiate : disabled
             admin-status : up
              oper-status : up
              admin-speed : 4x-sdr(10gbps)
               oper-speed : 4x-sdr(10gbps)
                link-trap : enabled
                phy-state : link-up

Code:

Topspin-33# show interface ib 1


================================================================================
                        InfiniBand Interface Information
================================================================================
                     port : 1
                     name : 1
                     type : ib4xTX
                     desc : 1 (65)
              last-change : none
                      mtu : 2048
 auto-negotiate-supported : no
           auto-negotiate : disabled
             admin-status : up
              oper-status : up
              admin-speed : 4x-sdr(10gbps)
               oper-speed : 4x-sdr(10gbps)
                link-trap : enabled
                phy-state : link-up

e100 · Aug 8, 2013

Do you have any nodes connected to the switches?

Do you have a subnet manager like open-sm running on a node that is connected to the switches?

RobFantini · Aug 8, 2013

e100 said:
Do you have any nodes connected to the switches?

Do you have a subnet manager like open-sm running on a node that is connected to the switches?

I have 5-6 nodes attached.

open-sm could be running on some nodes, I'll check and shut off.

I assume we do not want it installed ?

e100 · Aug 8, 2013

If you want to use the subnet manager in the switches you do not want opensm running on the nodes.

RobFantini · Aug 8, 2013

I shutdown and uninstalled subnet manager.

Then rebooted the switches.

But we still have this on both switches:

Code:

port-guid : 00:02:c9:02:00:27:26:b9

We also have a L3 Netgear switch . Maybe there interference from it... I'll check later.

In mean time if you think of something else please reply.

RobFantini · Aug 9, 2013

I have some conflicting data:

guid is not the same at em and http:

http:

Code:

http://10.1.10.32/main.html    guid 00:05:AD:00:00:02:24:43  , standby

http://10.1.10.33/main.html    guid 00:05:AD:00:00:02:78:FE  , standby

em:

Code:

Topspin-32# show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00
            subnet-prefix : fe:80:00:00:00:00:00:00
                  port-guid : 00:02:c9:03:00:07:4e:95
                     priority : 0
                  sm-state : master
                     sm-key : 00:00:00:00:00:00:00:00
                  act-count : 107296


Topspin-33# show ib sm sm-info subnet-prefix fe:80:00:00:00:00:00:00
            subnet-prefix : fe:80:00:00:00:00:00:00
                  port-guid : 00:02:c9:03:00:07:4e:95
                     priority : 0
                  sm-state : master
                     sm-key : 00:00:00:00:00:00:00:00
                  act-count : 107341

http shows mac addresses for the ethernet ports used on the switch as part of port-guid.

I asume the issue is caused by not doing a factory default reset before I started using the switch.

On one node, I've bonding set up on IB. I can unplug either IB cable and ping still works. so there is some highavailability working.

Next I'll try a factory reset, then reprogram. These switches are very easy to program compared to Layer 3 ethernet switches.

vcp_ai · Jan 8, 2014

Hi, thanks for all the info seen in this post, and sorry for re-using it, but may be it can help any other ...

Ive bougth a pair of 10Gb cards, and tested them in a 'pointo to point' method. It worked but was of no much real use.

Now I'w looking for a cheap (used) switch, and four more cards to test in a 'real' situation with Proxmox.

I've found this
IBM 73P3981 Topspin 120 Server Switch - Switch - 24 ports - InfiniBand - 4x Infi

ebay item 350970413637

You are talking about CISCO Top Spin 120..

Do you know if they are the same ??

Thanks in advance

e100 · Jan 8, 2014

Cisco purchased TopSpin in 2005: http://newsroom.cisco.com/dlls/2005/corp_041405.html?CMP=ILC-001

I believe that IBM sold Topspin products too because I have found documentation on their site related to the TopSpin 120.

Search

Search

infiniband configuration questions

e100

Renowned Member

RobFantini

Famous Member

e100

Renowned Member

RobFantini

Famous Member

e100

Renowned Member

RobFantini

Famous Member

e100

Renowned Member

RobFantini

Famous Member

e100

Renowned Member

e100

Renowned Member

RobFantini

Famous Member

e100

Renowned Member

RobFantini

Famous Member

e100

Renowned Member

RobFantini

Famous Member

RobFantini

Famous Member

vcp_ai

Renowned Member

e100

Renowned Member