Hello,
I am struggling with strange problem. I have HP DL360 G9 with HPE 544+FLR-QSFP network adapter (Mellanox Connect X 3 Pro). The card does not work under proxmox 8. The link is up, the lights are blinking, but no traffic at all. The only thing that is working is bonding (LACP). The PDU frames are sent and received. Nothing more.
Here are some details:
The strange thing is I cannot read details about qSFP+ module:
It does not matter if I will use 40G module, or 10G module in adapter, or breakout cable. The situation is always the same. There are some TX packets, no RX packets at all. Tried with Mikrotik/Cisco/Ufispace with always the same result - the card is not working:
I already tried to use variety of modules, including genuine Mellanox ones - nothing is working.
I made a test, on the same server, I've installed Centos 7, XCP-NG 8.2.1, and ESXi 6.7u3. On all of those systems the card is working well instantly on the same driver (4.0.0). When I switch to Proxmox - it is dead.
The installation of proxmox is clean,
PVE Packages version:
I am struggling with strange problem. I have HP DL360 G9 with HPE 544+FLR-QSFP network adapter (Mellanox Connect X 3 Pro). The card does not work under proxmox 8. The link is up, the lights are blinking, but no traffic at all. The only thing that is working is bonding (LACP). The PDU frames are sent and received. Nothing more.
Here are some details:
Bash:
root@pve:~# ethtool -i eno49
driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5700
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yesThe strange thing is I cannot read details about qSFP+ module:
Bash:
root@pve:~# ethtool -m eno49
netlink error: Invalid argument
Bash:
root@pve:~# dmesg -T| grep mlx
[Mon Apr 22 15:25:19 2024] mlx4_core: Mellanox ConnectX core driver v4.0-0
[Mon Apr 22 15:25:19 2024] mlx4_core: Initializing 0000:04:00.0
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0: DMFS high rate steer mode is: disabled performance optimized steering
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link)
[Mon Apr 22 15:25:25 2024] mlx4_en: Mellanox ConnectX HCA Ethernet driver v4.0-0
[Mon Apr 22 15:25:25 2024] mlx4_en 0000:04:00.0: Activating port:1
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 1: Using 32 TX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 1: Using 16 RX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 1: Initializing port
[Mon Apr 22 15:25:25 2024] mlx4_en 0000:04:00.0: registered PHC clock
[Mon Apr 22 15:25:25 2024] mlx4_en 0000:04:00.0: Activating port:2
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 2: Using 32 TX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 2: Using 16 RX rings
[Mon Apr 22 15:25:25 2024] mlx4_en: 0000:04:00.0: Port 2: Initializing port
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0 eno49d1: renamed from eth1
[Mon Apr 22 15:25:25 2024] <mlx4_ib> mlx4_ib_add: mlx4_ib: Mellanox ConnectX InfiniBand driver v4.0-0
[Mon Apr 22 15:25:25 2024] <mlx4_ib> mlx4_ib_add: counter index 2 for port 1 allocated 1
[Mon Apr 22 15:25:25 2024] <mlx4_ib> mlx4_ib_add: counter index 3 for port 2 allocated 1
[Mon Apr 22 15:25:25 2024] mlx4_core 0000:04:00.0 eno49: renamed from eth0
[Mon Apr 22 16:33:52 2024] mlx4_core 0000:04:00.0 eno49: entered allmulticast mode
[Mon Apr 22 16:33:52 2024] mlx4_en: eno49: Steering Mode 1
[Mon Apr 22 16:33:52 2024] mlx4_en: eno49: Link Up
[Mon Apr 22 16:35:07 2024] mlx4_core 0000:04:00.0 eno49: entered promiscuous mode
[Mon Apr 22 16:38:11 2024] mlx4_core 0000:04:00.0 eno49: left promiscuous modeIt does not matter if I will use 40G module, or 10G module in adapter, or breakout cable. The situation is always the same. There are some TX packets, no RX packets at all. Tried with Mikrotik/Cisco/Ufispace with always the same result - the card is not working:
Bash:
eno49: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 04:09:73:d3:b2:b1 txqueuelen 1000 (Ethernet)
RX packets 12027695 bytes 689682238 (657.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 34 bytes 2556 (2.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0I already tried to use variety of modules, including genuine Mellanox ones - nothing is working.
I made a test, on the same server, I've installed Centos 7, XCP-NG 8.2.1, and ESXi 6.7u3. On all of those systems the card is working well instantly on the same driver (4.0.0). When I switch to Proxmox - it is dead.
The installation of proxmox is clean,
PVE Packages version:
Bash:
proxmox-ve: 8.1.0 (running kernel: 6.5.13-5-pve)
pve-manager: 8.1.11 (running version: 8.1.11/cb1653fdade58a8b)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.5.13-5-pve-signed: 6.5.13-5
proxmox-kernel-6.5: 6.5.13-5
proxmox-kernel-6.5.11-8-pve-signed: 6.5.11-8
ceph-fuse: 17.2.7-pve2
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.3
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.1.2
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.0.6
libpve-network-perl: 0.9.6
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.0
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.5-1
proxmox-backup-file-restore: 3.1.5-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.1.5
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.1.5
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.4
pve-firmware: 3.11-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.1
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.1.4
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2