Stumped - NIC Issues?

[Brandon Brown]

Hello! I'm trying to cluster 3 servers together. I have the OS installed and nothing has been modified yet other than the NIC settings. Please read through, but start with the basics... I'm trying to ditch the LACP idea and see if I can get a single 10gbps port working on Server 1 and 3

Hardware
Servers - 3 x HP Proliant 380 Gen7s
NICs - 2 port CNA (Mellanox I believe, maybe HP branded)
Server #2 has a 4 x Gig-E card and a 2 x Gig E card. I wanted to bond 8 of these to get 8gbps hopefully for the Ceph network. As I've been reading on, I don't know if that's possible.

Switches
2 x Cisco 93180s vPC'd
VLAN 599 for Ceph

Here's what I was initially trying to accomplish:

• Setup server 1 and 3's 2 x 10gbps NICs as LACP and vPC on the Nexus end
• Single purpose is for the Ceph network (VLAN 599)

Here's what I initially configured and what I saw on both server/switch side:
**please assume the commands entered were correct - typing this manually**

Server config...
enp13s0f0 and enp13s0f0 are inet manual like the rest of the ports
iface bond0 inet static
slaves enp13s0f0 enp13s0f1
address 10.0.0.91
netmask 255.255.255.0
bond_mode 802.3ad
bond_miimon 100
bond_xmit_hash_policy layer2+3 <-- also tried 3+4

Switch config (same on both switch A and switch B)...
interface port-channel91
description blabvm1 ceph-net
switchport
switchport access vlan 599
spanning-tree port type edge
vpc 91

interface port-channel93
description blabvm3 ceph-net
switchport
switchport access vlan 599
spanning-tree port type edge
vpc 93

interface Ethernet1/5
description blabvm1 ceph-net
switchport
switchport access vlan 599
spanning-tree port type edge
channel-group 91 mode active
no shutdown

interface Ethernet1/7
description blabvm3 ceph-net
switchport
switchport access vlan 599
spanning-tree port type edge
channel-group 93 mode active
no shutdown

Note that these are in access mode...
93180A(config-if)# show int eth1/5 switchport
Name: Ethernet1/5
Switchport: Enabled
Operational Mode: access
Access Mode VLAN: 599 (ceph-cluster)

• Ports DOWN on the server
• Ports are LACP Suspended (No PDUs received) on the switch side

**Now that is all said... let's start with the basics and what I have setup now**

10gbps port config
auto enp13s0f0
iface enp13s0f0 inet static
address 10.0.0.91
netmask 255.255.255.0

#ip a output (note it is 'state DOWN')

6: enp13s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 00:00:c9:85:0f:7e brd ff:ff:ff:ff:ff:ff inet 10.0.0.91/24 brd 10.0.0.255 scope global enp13s0f0 valid_lft forever preferred_lft forever

Switch A interface config and stats and mode...

93180A(config-if)# show run int eth1/5

!Command: show running-config interface Ethernet1/5
!Time: Fri Mar 16 20:06:01 2018

version 7.0(3)I6(1)

interface Ethernet1/5
description blabvm1 ceph-net
switchport
switchport access vlan 599
spanning-tree port type edge
no shutdown

93180A(config-if)# show mac address-table int eth1/5
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 599 0000.c985.0f7d dynamic 0 F F Eth1/5


93180A(config-if)# show int eth1/5 switchport
Name: Ethernet1/5
Switchport: Enabled
Operational Mode: access
Access Mode VLAN: 599 (ceph-cluster)


93180A(config-if)# show int eth1/5
Ethernet1/5 is up
admin state is up, Dedicated Interface
Hardware: 100/1000/10000/25000 Ethernet, address: 500f.8013.d28c (bia 500f.8013.d28c)
Description: blabvm1 ceph-net
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is access
full-duplex, 10 Gb/s, media type is 10G
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 12:43:08
Last clearing of "show interface" counters 12:57:24
10 interface resets
30 seconds input rate 177568 bits/sec, 346 packets/sec
30 seconds output rate 504 bits/sec, 0 packets/sec
Load-Interval #2: 5 minute (300 seconds)
input rate 177.31 Kbps, 346 pps; output rate 264 bps, 0 pps
RX
0 unicast packets 16162771 multicast packets 0 broadcast packets
16162787 input packets 1034455720 bytes
0 jumbo packets 0 storm suppression bytes
38 runts 0 giants 0 CRC 0 no buffer
38 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 0 input discard
0 Rx pause
TX
0 unicast packets 25648 multicast packets 0 broadcast packets
25648 output packets 2162284 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause



THANK YOU very much for taking the time to read through this all and help out. Let me know what other output from the server would be useful and if there are any other troubleshooting commands that I should use to figure out.

I am only attempting to ping server1 from server 3 or vice versa. I'd like to at least get that far. Pings time out, even when I specify the interface.

root@blabvm1:~# ping 10.0.0.93 -I enp13s0f0
PING 10.0.0.93 (10.0.0.93) from 10.0.0.91 enp13s0f0: 56(84) bytes of data. --- 10.0.0.93 ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8195ms

 

[Brandon Brown]

One thing I just noticed is that the MAC address reported on the switches are "2" hex digits lower than what the 'ip a' output shows. For all 4 x 10 gbps ports.

 

[Brandon Brown]

Any ideas? I kept circling back to it being a driver issue.

It's using ixgbe as the driver
Seems like it should be using be2net
Ethtool is reporting one port being TP (twisted pair) and the other as Fibre. I have passive copper TwinAx plugged in to both ports. So not exactly what you'd call twisted pair.

Tried forcin speed and duplex on both sides .
Ethtool didn't like me trying to change speed duplex or autonegotiation at one point.
I read that LRO and GRO needed turned off .I built the ixgbe driver 5.3.6 from source with a flag to turn off LRO. Looked like that worked based on the output of ethtool or modinfo, can't remember which one.