Mellanox ConnectX-4 issue since 6.5.11-8

[vincentp]

Hi

I have 2 pve servers (cluster) which I updated to 6.5.11-8 and rebooted yesterday. Since then, one of the two mellanox adapters stopped functioning normally.

Each server has 2 ConnectX-4 Lx adapters

1 x MCX4121A-ACA_Ax PCIE x8 - FW 14.32.1010 - Not working
1 x MCX4621A-ACA_Ax OCP 3.0 - FW 14.32.1010 - Working

I updated the firmware yesterday hoping it would help with the issue but it made no difference.

I don't remember the exact kernel version we were running before but the last reboot (which would have been for an update) was Dec 19th. Unfortunatley I stupidly cleaned up unused packages before I knew about this issue so rebooting to previous kernal not tried yet (need to learn how to downgrade).

The servers are linked directly via DAC cable to each other and the link shows as UP, however no traffic is flowing and I cannot ping the other server. This was working fine before I rebooted after updating yesterday. No other configuration changes have been made (apart from the fw update). This link is used for replication & migration - so whilst not critical (backups are still working over other links) I would like to avoid reboots where possible (production websites).

Server A

root@galaxy:~# ethtool -i enp161s0f0np0
driver: mlx5_core
version: 6.5.11-8-pve
firmware-version: 14.32.1010 (MT_2420110034)
expansion-rom-version:
bus-info: 0000:a1:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

root@galaxy:~#ip a
9: enp161s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ec:0d:9a:2d:ea:ca brd ff:ff:ff:ff:ff:ff
    inet 10.0.125.100/24 scope global enp161s0f0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::ee0d:9aff:fe2d:eaca/64 scope link
       valid_lft forever preferred_lft forever

Server B

root@raptor:~# ethtool -i enp129s0f0np0
driver: mlx5_core
version: 6.5.11-8-pve
firmware-version: 14.32.1010 (MT_2420110034)
expansion-rom-version:
bus-info: 0000:81:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

root@raptor:~# ip a
9: enp129s0f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether ec:0d:9a:d4:95:c4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.125.110/24 scope global enp129s0f0np0
       valid_lft forever preferred_lft forever
    inet6 fe80::ee0d:9aff:fed4:95c4/64 scope link
       valid_lft forever preferred_lft forever

I guess it could be a bad cable - I'm waiting on the DC staff to pull their finger out and try reseating the cable and I shipped a new cable to the DC (should arrive today) just in case - but the timing of the issue (after an update/reboot) suggests to me it's something else.

Any suggestions on what else I can check? I am not a linux expert and relatively new to proxmox.

 

[vincentp]

More info

root@raptor:~# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.2.16-20-pve
6.5.11-7-pve
6.5.11-8-pve

So I guess 6.5.11-7-pve was the previous kernel version.

 

[vincentp]

So weirdly, this turned out to be a DAC cable issue. Swapped out for a new cable and it works fine. Very odd that it failed after an update/reboot