Mellanox ConnectX-3 Issues

[crainbramp]

Hello:

Running the latest Proxmox, have a ConnectX-3 dual port card in the server running as Ethernet. Cannot seem to bring up any interfaces on it. The bridge comes up, but doesn't assign any VLAN interfaces / IPs to it. Seems to bomb out at this point.

Firmware is the latest: 2.42.5000

Below are some output and some of the syslog. Would appreciate it if anyone could give an assist:

root@node03sea:~# pveversion
pve-manager/6.0-4/2a719255 (running kernel: 5.0.15-1-pve)

03:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]
        Subsystem: Mellanox Technologies MT27500 Family [ConnectX-3]
        Kernel driver in use: mlx4_core
        Kernel modules: mlx4_core
root@node03sea:~# lspci -vv -s 03:00.0 | grep "Part number" -A 3
                        [PN] Part number: MCX312A-XCBT
                        [EC] Engineering changes: A5
                        [SN] Serial number: MT1338X00222
                        [V0] Vendor specific: PCIe Gen3 x8

root@node03sea:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 2c:59:e5:41:f9:c0 brd ff:ff:ff:ff:ff:ff
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 2c:59:e5:41:f9:c1 brd ff:ff:ff:ff:ff:ff
4: eno3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 2c:59:e5:41:f9:c2 brd ff:ff:ff:ff:ff:ff
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 2c:59:e5:41:f9:c3 brd ff:ff:ff:ff:ff:ff
6: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000
    link/ether f4:52:14:0b:c6:50 brd ff:ff:ff:ff:ff:ff
7: ens2d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether f4:52:14:0b:c6:51 brd ff:ff:ff:ff:ff:ff
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 2c:59:e5:41:f9:c3 brd ff:ff:ff:ff:ff:ff
9: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f4:52:14:0b:c6:50 brd ff:ff:ff:ff:ff:ff

auto vmbr1
iface vmbr1 inet manual
        bridge_ports ens2
        bridge_stp off
        bridge_fd 0
        bridge_vlan-aware yes

auto vmbr1.10
iface vmbr1.10 inet static
        address  10.0.44.243
        netmask  24

auto vmbr1.20
iface vmbr1.20 inet static
        address  10.0.45.243
        netmask  24

Dec 17 13:35:08 node03sea systemd[1]: Started Proxmox VE Login Banner.
Dec 17 13:35:08 node03sea ifup[1058]: Waiting for vmbr0 to get ready (MAXWAIT is 2 seconds).
Dec 17 13:35:08 node03sea kernel: [   13.183604] mlx4_en: ens2d1: Link Up
Dec 17 13:35:08 node03sea kernel: [   13.238430] mlx4_en: ens2: Link Up
Dec 17 13:35:08 node03sea systemd-udevd[658]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 17 13:35:08 node03sea systemd-udevd[658]: Could not generate persistent MAC address for vmbr1: No such file or directory
Dec 17 13:35:08 node03sea kernel: [   13.313321] vmbr1: port 1(ens2) entered blocking state
Dec 17 13:35:08 node03sea kernel: [   13.313324] vmbr1: port 1(ens2) entered disabled state
Dec 17 13:35:08 node03sea kernel: [   13.313415] device ens2 entered promiscuous mode
Dec 17 13:35:08 node03sea kernel: [   13.329335] mlx4_en: ens2: Steering Mode 1
Dec 17 13:35:08 node03sea ifup[1058]: Waiting for vmbr1 to get ready (MAXWAIT is 2 seconds).
Dec 17 13:35:08 node03sea kernel: [   13.477898] device ens2 left promiscuous mode
Dec 17 13:35:08 node03sea ifup[1058]: RTNETLINK answers: No space left on device
Dec 17 13:35:08 node03sea ifup[1058]: run-parts: /etc/network/if-up.d/bridgevlan exited with return code 255
Dec 17 13:35:08 node03sea ifup[1058]: ifup: failed to bring up vmbr1
Dec 17 13:35:08 node03sea ifup[1058]: Waiting for vmbr1 to get ready (MAXWAIT is 2 seconds).
Dec 17 13:35:08 node03sea ifup[1058]: RTNETLINK answers: No space left on device
Dec 17 13:35:08 node03sea ifup[1058]: run-parts: /etc/network/if-up.d/bridgevlan exited with return code 255
Dec 17 13:35:08 node03sea ifup[1058]: ifup: failed to bring up vmbr1
Dec 17 13:35:08 node03sea ifup[1058]: ifup: could not bring up parent interface vmbr1
Dec 17 13:35:08 node03sea ifup[1058]: Waiting for vmbr1 to get ready (MAXWAIT is 2 seconds).
Dec 17 13:35:08 node03sea ifup[1058]: RTNETLINK answers: No space left on device
Dec 17 13:35:08 node03sea ifup[1058]: run-parts: /etc/network/if-up.d/bridgevlan exited with return code 255
Dec 17 13:35:08 node03sea ifup[1058]: ifup: failed to bring up vmbr1
Dec 17 13:35:08 node03sea ifup[1058]: ifup: could not bring up parent interface vmbr1
Dec 17 13:35:08 node03sea systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Dec 17 13:35:08 node03sea systemd[1]: networking.service: Failed with result 'exit-code'.
Dec 17 13:35:08 node03sea systemd[1]: Failed to start Raise network interfaces.

 

[bobblehead]

The card only supports ~128 VLANs; see the "No space left on device" error message. Don't use the VLAN aware settings and you'll get further.

 

[crainbramp]

A little late, but thanks. Removing the offending section worked.

Also, a note for others: I had found the information concerning the 128 VLAN limit, and had changed my network/interfaces to include "bridge_vids 2-128" and this did not work. I simply assumed something else was wrong. It appears you have to remove this entire descriptor.