Help with ProxMox OpenFabric Configuration

[vnick_coty]

Hello, everyone,
I've got a cluster of four ProxMox VE systems that I use a test stack. The systems are Cisco UCS B blades connected to redundant UCS Fabric Interconnections.

I've got a couple of vNICs assigned to the UCS systems that I'm trying to use to create an Openfabric configuration. The vNICs have a native VLAN assigned (no tagging) and jumbo frames enabled. It's a non-routed VLAN - strictly layer 2.

The issue I'm currently having is that, when I create the OpenFabric configuration, the nodes never "see" each other. Here are the steps I've taken:

• Verify that the vNICs can actually communicate with each other by assigning them IP addresses on the same subnet and pinging between the nodes.
• Remove the IP addresses, and create a new OpenFabric configuration.
  • Give the fabric a name: us1
  • Assign the fabric an IPv4 subnet: 192.168.148.0/24
  • Add each node to the fabric:
    • Select each node (node1, etc.)
    • Type an IP address to identify the node on the fabric: 192.168.148.10 (etc.)
    • Select the vNIC to use (enp103s0f1)
    • "Create Another" until all four nodes are added.
  • Apply the SDN configuration.

What happens, then, is that the OpenFabric shows up in the SDN configuration, but it shows up as "Not OK." I've done a few things to troubleshoot:

• Launch a shell on the nodes and run vtysh
  • run"show openfabric neighbors"
    • No neighbors are shown
• Run tcpdump on one of the nodes on the interface selected:
  • tcpdump -i enp103s0f1
  • I see the IS-IS, p2p IIH packets coming from all four of the nodes, but there are never any responses/replies - just continuous IIH packets.
• Run vtysh and enable debugging for OpenFabric:
  • log daemon fabricd file /tmp/fabricd.log debugging
  • debug openfabric adj-packets
    • Repeat for each of the logging facilities except packet-dump
  • I've examined the log file, and the only messages I see are the ones stating that the current node is sending out it's Hello/Adjacency packet - it never seems to acknowledge that it's seen anything from any other host.
• Check the PVE firewall configuration to make sure no rules are enabled that would block the traffic - the firewall is disabled at the moment.

Any ideas what else I can check to figure out why the systems don't seem to be acknowledging that they've received the IIH packets or doing anything about it?

 

[shanreich]

Are all network cards of the nodes connected to the same switch / VLAN? OpenFabric assumes that links are point-to-point only, so it will not work if all of them share the same layer-2 domain.