[SOLVED] 10gb direct ethernet connection between Proxmox and FreeNAS

[plb2010]

I am trying to create a direct ethernet connection between a Proxmox host and a FreeNAS box using 2 Mellanox 10gb NICs and am unable to get the two systems to see (ping) one another over the direct connection. (They can see each other over my LAN using the onboard NICs on each box. Ping, mounting NFS shares, etc all work fine.) My goal is, using the 10gb connection, to make several NFS shares from the FreeNAS box available via bind mounts to containers in the Proxmox host.

The problem “seems” to be on the Proxmox host as FreeNAS sees the Mellanox card, allows me to ping it locally (but not ping the Proxmox box), shows the driver loading at start-up, etc. On the Proxmox side, the card shows up in the Host / System / Network panel but when I try to ping its assigned address from within Proxmox, it fails and if I try to ping the FreeNAS's 10gb address, it fails as well.

The lights on the back of each card are green and they are connected with a proven-to-work Mellanox MC3309130-001 1m. DAC cable. These are ethernet only cards, so there is also no issue of needing to set the ports to ethernet mode.

What am I overlooking or misunderstanding?

Details:
BOX A: Proxmox 6.1-3

• 2 onboard 1g NICs (plus a third nic for the IPMI/BNC). Only one of the NICs is configured at 192.168.1.11/24 and connects to my router at 192.168.1.1
• 1 Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet @10.0.0.2/32 with no gateway configured.

Box B: FreeNAS 11.2-U7

• 2 onboard 1g NICs (plus a third nic for the IPMI/BNC). Only one of the NICs is configured @ 192.168.1.6/24 and connects to my router at 192.168.1.1
• 1 Mellanox MCX311A-XCAT CX311A ConnectX-3 EN 10G Ethernet @ 10.0.0.1/32 with no gateway configured.

From the Proxmox shell...

root@proxmox01:~# lspci -v | grep Mellanox

02:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

        Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]



root@proxmox01:~# ifconfig enp0s20f0u8u2c2

enp0s20f0u8u2c2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

        ether be:c1:d5:3c:f6:d5  txqueuelen 1000  (Ethernet)

        RX packets 0  bytes 0 (0.0 B)

        RX errors 0  dropped 0  overruns 0  frame 0

        TX packets 269  bytes 25627 (25.0 KiB)

        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



root@proxmox01:~# ip link show

5: enp0s20f0u8u2c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000

    link/ether be:c1:d5:3c:f6:d5 brd ff:ff:ff:ff:ff:ff





root@proxmox01:~# dmesg | grep mlx4

[    1.753589] mlx4_core: Mellanox ConnectX core driver v4.0-0

[    1.753602] mlx4_core: Initializing 0000:02:00.0

[    1.753650] mlx4_core 0000:02:00.0: enabling device (0000 -> 0002)

[    8.995791] mlx4_core 0000:02:00.0: DMFS high rate steer mode is: disabled performance optimized steering

[    8.996124] mlx4_core 0000:02:00.0: 31.504 Gb/s available PCIe bandwidth (8 GT/s x4 link)

[    9.057635] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0

[    9.057794] mlx4_en 0000:02:00.0: Activating port:1

[    9.060449] mlx4_en: 0000:02:00.0: Port 1: Using 16 TX rings

[    9.060450] mlx4_en: 0000:02:00.0: Port 1: Using 16 RX rings

[    9.060774] mlx4_en: 0000:02:00.0: Port 1: Initializing port

[    9.061214] mlx4_en 0000:02:00.0: registered PHC clock

[    9.061368] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0

[    9.061962] mlx4_core 0000:02:00.0 eno1: renamed from eth0

[    9.065802] <mlx4_ib> mlx4_ib_add: counter index 1 for port 1 allocated 1

[   10.390429] mlx4_en: eno1: Link Up



root@proxmox01:~# nano /etc/network/interfaces

auto lo

iface lo inet loopback



iface eno1 inet static

        address  10.0.0.2

        netmask  32



iface eno2 inet manual



auto enp0s20f0u8u2c2

iface enp0s20f0u8u2c2 inet static

        address  10.0.0.2

        netmask  32



iface eno3 inet manual



auto vmbr0

iface vmbr0 inet static

        address  192.168.1.11

        netmask  24

        gateway  192.168.1.1

        bridge-ports eno2

        bridge-stp off

        bridge-fd 0

 

[syfy323]

Are you sure it works this way with /32?
I would expect a small subnet or the "pointopoint" (no typo) directive pointing to the other node.

 

[syfy323]

Ok, I am confused, you miss the "auto" directives and have your IP duplicated, that can't work.
According to your log, eno1 is your mlx adapter.

 

[plb2010]

Are you sure it works this way with /32?
I would expect a small subnet or the "pointopoint" (no typo) directive pointing to the other node.

No. I started with .../24 but after I started running into walls ended up going down the rabbit hole of over-googled-advise from other forums and so tried using netmask from 24-32 and nothing worked.

I don't know about the pointopoint... reading...

 

[plb2010]

Ok, I am confused, you miss the "auto" directives and have your IP duplicated, that can't work.
According to your log, eno1 is your mlx adapter.

Perfect. Thanks. The interface was originally showing up as enp0s20f0u8u2c2 but after a recent reboot switched to eno1, but did not notice--it was the sudden appearance of eno3 that stumped me. I made the switch and things are working.

From PVE:

root@proxmox01:~# ping -c 3 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.165 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.718 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.686 ms

--- 10.0.0.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 56ms
rtt min/avg/max/mdev = 0.165/0.523/0.718/0.253 ms

From FreeNAS:
.... cannot seem to copy & paste from the FreeNAS shell but... it works.

Thanks so much. I've been pulling my hair out over this for three days! :/

 

[plb2010]

Dumb question but... how do I mark the thread as solved?

 

[syfy323]

Dumb question but... how do I mark the thread as solved?

I'm not exactly sure but can you edit the title? There should be a drop-down for that.

 

[syfy323]

From FreeNAS:
.... cannot seem to copy & paste from the FreeNAS shell but... it works.

Thanks so much. I've been pulling my hair out over this for three days! :/

Thats why I never use predictive network interfaces and switched to the old behavior (params: biosdevname=0 net.ifnames=0).
I know why they are called predictive but actually, most automation tools work better with the old naming scheme.