HPE 544+FLR-QSFP not working on Proxmox 8

[dymek]

Hello,

I am struggling with strange problem. I have HP DL360 G9 with HPE 544+FLR-QSFP network adapter (Mellanox Connect X 3 Pro). The card does not work under proxmox 8. The link is up, the lights are blinking, but no traffic at all. The only thing that is working is bonding (LACP). The PDU frames are sent and received. Nothing more.

Here are some details:

root@pve:~# ethtool -i eno49
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5700
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

The strange thing is I cannot read details about qSFP+ module:

root@pve:~# ethtool -m eno49
netlink error: Invalid argument

root@pve:~# dmesg -T| grep mlx
[Mon Apr 22 15:25:19 2024] mlx4_core: Mellanox ConnectX core driver v4.0-0
[Mon Apr 22 15:25:19 2024] mlx4_core: Initializing 0000:04:00.0
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[Mon Apr 22 15:25:25 2024] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[Mon Apr 22 15:25:25 2024] mlx4_en 0000:04:00.0: Activating port:1
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 1: Using 32 TX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 1: Using 16 RX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 1: Initializing port
[Mon Apr 22 15:25:25 2024] mlx4_en 0000:04:00.0: registered PHC clock
[Mon Apr 22 15:25:25 2024] mlx4_en 0000:04:00.0: Activating port:2
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 2: Using 32 TX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 2: Using 16 RX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 2: Initializing port
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0 eno49d1: renamed from eth1
[Mon Apr 22 15:25:25 2024] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[Mon Apr 22 15:25:25 2024] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[Mon Apr 22 15:25:25 2024] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0 eno49: renamed from eth0
[Mon Apr 22 16:33:52 2024] mlx4_core 0000:04:00.0 eno49: entered allmulticast mode
[Mon Apr 22 16:33:52 2024] mlx4_en: eno49: Steering Mode 1
[Mon Apr 22 16:33:52 2024] mlx4_en: eno49: Link Up
[Mon Apr 22 16:35:07 2024] mlx4_core 0000:04:00.0 eno49: entered promiscuous mode
[Mon Apr 22 16:38:11 2024] mlx4_core 0000:04:00.0 eno49: left promiscuous mode

It does not matter if I will use 40G module, or 10G module in adapter, or breakout cable. The situation is always the same. There are some TX packets, no RX packets at all. Tried with Mikrotik/Cisco/Ufispace with always the same result - the card is not working:

eno49: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 04:09:73:d3:b2:b1  txqueuelen 1000  (Ethernet)
        RX packets 12027695  bytes 689682238 (657.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 34  bytes 2556 (2.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I already tried to use variety of modules, including genuine Mellanox ones - nothing is working.

I made a test, on the same server, I've installed Centos 7, XCP-NG 8.2.1, and ESXi 6.7u3. On all of those systems the card is working well instantly on the same driver (4.0.0). When I switch to Proxmox - it is dead.

The installation of proxmox is clean,

PVE Packages version:

proxmox-ve: 8.1.0 (running kernel: 6.5.13-5-pve)
pve-manager: 8.1.11 (running version: 8.1.11/cb1653fdade58a8b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.1.2
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.0
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.4
pve-firmware: 3.11-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2

 

[floh8]

Mellanox give no support for Connect3 for Debian 12. u can switch back to Debian 11 ->pve 7 or buy connect 4

 

[dymek]

Well I know that, but the strangest thing is - dell branded card is working on R730xd server. I have connect-x 5 but still I want to use internal LOM and 40/100G network connection.

EDIT:

Very strange, but I've made it to work. The card works with direct address or vlan on it, when I am adding it to Linux Bridge - it stops working. So I've installed open-vswitch, and add it to open-vswitch bridge. The card is working on OVS. Seems like this is a bug in Linux Bridge...

 

[zzz09700]

Some network cards don't support bridging, some manufacturer purposely made strange changes into the bios of the card and their MB so full functionality is only available when paired with the mb from the same brand.

 

[dymek]

So this is HP Branded FlexLOM installed on HP server. The vendor is not a problem here. The same card works with bridge with kernel 5.x, but it is not working with 6.x