Mellanox ConnectX-3 & Proxmox 9.1

[Vampire337]

Background: Last summer I did some searching for a compatible fiber-channel solution to connect my two virtual host servers (Dell R420) to my TrueNAS server (Dell R630XD). Reviews seemed to suggest the ConnectX-3 card was a good fit that won't break the bank, so I bought 3 in August. I quickly learned that the QSFC cards are not physically compatible with the SFC modules I bought, so I ended up buying some patch cables and got it plugged in. At the time I was running PVE 8.3 on my primary server, and I followed the steps in the OP of https://forum.proxmox.com/threads/h...ox-connectx-3-cards-for-sriov-and-vfs.121927/. I installed the older Mellanox drivers like it says. I created a couple ZFS mirrored arrays in TrueNAS, shared them via NFS, and mapped them in Proxmox. I migrated VMs to the new datastores, and everything seemed to work just fine.

A couple months later I upgraded to PVE 9, and that's about when I noticed (Alpine Linux) VMs running on the NFS storage sometimes refuse to reboot or shutdown. If given a PVE GUI command to Shutdown or Reboot (not Reset or Stop), sometimes VMs would just hang until I issue a Stop (or reboot the PVE node). The same has been true about issuing a "reboot" or "poweroff" command from the console or SSH terminal session, and the console just hangs. I had thought this might be related to something I'm doing with backups, but troubleshooting has me wondering if it's related to my ConnectX-3 drivers and the upgrade-in-place from 8.3.

Over the holidays I was finally able to remove ESXi from my secondary virtual host, and I installed PVE 9. I tried the steps from that forum post and nothing worked. I tried the suggestions by others, and even posted my feedback (so far no one has responded). I've even considered installing PVE 8.3, installing the ConnectX-3 drivers per OP, then upgrading to the latest PVE, but I'm concerned that might be what's causing my VM halting issues. I tried bouncing the question off ChatGPT and it suggested an incompatibility between the older Mellanox driver and the kernel in PVE 9.

I have two questions: First, is there a best-practice method for installing the Mellanox ConnectX-3 card in PVE 9? Where do I find the most compatible drivers to get my system back to stable? Second, did I make a mistake in going with ConnectX-3 and it's just not supportable in PVE going forward? If so, what's a better high speed link hardware to use for segmenting my virtual disk traffic apart from my regular subnets? My home lab is a testing ground to prove out what I'm recommending for customers in industrial automation (SCADA platforms), so I want to get something reliable & supportable to show alternatives to HyperV as they're migrating away from VMWare platforms.

 

[spirit]

native kernel driver should work, no need to install drivers from mellanox.

But connect3x are pretty old and they use a different driver than connectx4,5,6... so I don't known if they are still working fine. (I don't have used them since 6~7years now)

I still have connectx4,5,6 in prod with same drivers wihtout problem, works out of the box.

 

[Vampire337]

native kernel driver should work, no need to install drivers from mellanox.

But connect3x are pretty old and they use a different driver than connectx4,5,6... so I don't known if they are still working fine. (I don't have used them since 6~7years now)

I still have connectx4,5,6 in prod with same drivers wihtout problem, works out of the box.

The ones I installed in my 2 servers do not natively appear anywhere in Proxmox, either when installed into an existing PVE 8.3 or installed before a fresh install of PVE 9.1. I wish it were as simple as plug-and-play and let the native kernel do its thing.

Maybe ConnectX-4 is the min I should've gone with. Thanks for the feedback!

 

[santiagobiali]

I've been using Mellanox ConnectX on PVE 7~9 and never had a problem. It showed up every time and worked.
Drivers are included into Proxmox kernel, never had to install custom drivers.

root@p99:~# lspci | grep -i mellanox
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

root@p99:~# lsmod | grep -i mlx
mlx4_en               159744  0
mlx4_core             413696  1 mlx4_en

cat /etc/modprobe.d/mlx-infiniband.conf
blacklist mlx4_ib

root@p99:~# grep -i mlx4 /boot/config-6.17.9-1-pve
CONFIG_MLX4_EN=m
CONFIG_MLX4_EN_DCB=y
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
CONFIG_MLX4_INFINIBAND=m

 

[alexskysilk]

The ones I installed in my 2 servers do not natively appear anywhere in Proxmox,

Some things to consider:

1. Connectx3 cards were often shipped as VPI cards. what that means is that each port can operated as ethernet or IB. When set to auto mode (default) they will default to IB when no link is detected at boot. You can and should force the ports to the operating mode you want to employ.
2. not all mlx4 modules load by default. if they do now, you will need to create a modules.conf file and include the modules required for your desired mode of operation.

https://docs.nvidia.com/networking/...NpbmdtbHhjb25maWd0b1NldElCL0VUSFBhcmFtZXRlcnM

 

[Vampire337]

I've been using Mellanox ConnectX on PVE 7~9 and never had a problem. It showed up every time and worked.
Drivers are included into Proxmox kernel, never had to install custom drivers.

root@p99:~# lspci | grep -i mellanox
05:00.0 Ethernet controller: Mellanox Technologies MT27500 Family [ConnectX-3]

root@p99:~# lsmod | grep -i mlx
mlx4_en               159744  0
mlx4_core             413696  1 mlx4_en

cat /etc/modprobe.d/mlx-infiniband.conf
blacklist mlx4_ib

root@p99:~# grep -i mlx4 /boot/config-6.17.9-1-pve
CONFIG_MLX4_EN=m
CONFIG_MLX4_EN_DCB=y
CONFIG_MLX4_CORE=m
CONFIG_MLX4_DEBUG=y
CONFIG_MLX4_CORE_GEN2=y
CONFIG_MLX4_INFINIBAND=m

Thanks for the commands. As a sanity check, I wiped my secondary server and installed 9.1 fresh, and as soon as I connected I ran the same commands. I swear it wasn't there before, but lspci and the last grep commands return the same as yours. I'll chalk this up to user error and/or ignorance. I do see that where your card returns "mlx_en", mine returns "mlx_ib" entries. So it seems my issue might just be as simple as needing to configure the card to ethernet.

Some things to consider:

1. Connectx3 cards were often shipped as VPI cards. what that means is that each port can operated as ethernet or IB. When set to auto mode (default) they will default to IB when no link is detected at boot. You can and should force the ports to the operating mode you want to employ.
2. not all mlx4 modules load by default. if they do now, you will need to create a modules.conf file and include the modules required for your desired mode of operation.

https://docs.nvidia.com/networking/...NpbmdtbHhjb25maWd0b1NldElCL0VUSFBhcmFtZXRlcnM

For forcing the mode, I'm assuming that's a function of mlxconfig, but to confirm, can I do that by modifying modules.conf alone? I checked on my primary PVE and there's currently nothing in /etc/modules.conf. What should I need to put there?

Since the base PVE install does not include mlxconfig, I tried running through the original steps for downloading & installing mft-4.22.1-11-x86_64-deb.tgz and I got hit with "Package 'linux-headers' has no installation candidate". Assuming I need mlxconfig, how do I go about installing that prerequisite?

 

[santiagobiali]

your card returns "mlx_en", mine returns "mlx_ib" entries

Yup, your card is in infiniband mode, just need to force it to ethernet mode.

I don't remember the exact procedure I took to force it, but this should help you installing mft:

P

Thread 'How-to: configure Mellanox ConnectX-3 cards for SRIOV and VFS'

Jan 31, 2023

Hey folks,

Got real darn tired of trying to poke and prod and guess and then forget a month later what i did in the first place on a new server so im including my guide to get your mellanox connectx-3 card up and running on proxmox 7.3

Ultimately it looks like the real problem/struggle I dealt with that wasn't part of the official nvidia guides was two things, 1) that im using the OS version of the drivers instead of OFED which obviously on proxmox 7.3 and debian 11 is not supported and never will be now. Then 2) That the kernel is only loading the mlx4_core module and neither mlx4_en or...

• pinkyd
• Replies: 23
• Forum: Proxmox VE: Installation and configuration

• 

I got hit with "Package 'linux-headers' has no installation candidate"

# Choose the kernel version you are using and install only it.

#PVE 9
apt install proxmox-headers-6.14
apt install proxmox-headers-6.17

# PVE 8
apt install proxmox-headers-6.2
apt install proxmox-headers-6.5
apt install proxmox-headers-6.8
apt install proxmox-headers-6.11
apt install proxmox-headers-6.14

I blacklisted ib modules in my proxmox to troubleshoot some stuff a while back, but it's not needed.

cat /etc/modprobe.d/mlx-infiniband.conf
blacklist mlx4_ib

 

[Vampire337]

Yes, that thread is the one I linked in my OP. It has you download & install mft-4.22.1-11-x86_64-deb.tgz, which worked great on my PVE 8.3 install (and has since been upgraded). It just wasn't working out for the fresh PVE 9.1 install.

Thanks for the proxmox-headers installs. I am on 6.17 so I picked that one, and it got me... further.

# cd mft-4.22.1-11-x86_64-deb/
# ./install.sh
-I- Removing mft external packages installed on the machine
-I- Installing package: /root/mft-4.22.1-11-x86_64-deb/SDEBS/kernel-mft-dkms_4.22.1-11_all.deb
-E- Failed to install package "/root/mft-4.22.1-11-x86_64-deb/SDEBS/kernel-mft-dkms_4.22.1-11_all.deb"', for more details see /tmp/mft.220629.logs/install_kernel-mft-dkms_4.22.1-11_all.deb.log

# cat /tmp/mft.220629.logs/install_kernel-mft-dkms_4.22.1-11_all.deb.log

Error! Bad return status for module build on kernel: 6.17.9-1-pve (amd64)
Consult /var/lib/dkms/kernel-mft-dkms/4.22.1/build/make.log for more information.
dpkg: error processing package kernel-mft-dkms (--install):
 installed kernel-mft-dkms package post-installation script subprocess returned error exit status 10
Errors were encountered while processing:
 kernel-mft-dkms
 only for 6.17.9-1-pve
Building for architecture amd64
Building initial module for 6.17.9-1-pve

# cat /var/lib/dkms/kernel-mft-dkms/4.22.1/build/make.log
DKMS (dkms-3.2.2) make.log for kernel-mft-dkms/4.22.1 for kernel 6.17.9-1-pve (amd64)
Thu Feb 12 12:27:33 PM EST 2026

Building module(s)
# command: make -j24 KERNELRELEASE=6.17.9-1-pve all KPVER=6.17.9-1-pve
cd mst_backward_compatibility/mst_pci && make
make[1]: Entering directory '/var/lib/dkms/kernel-mft-dkms/4.22.1/build/mst_backward_compatibility/mst_pci'
make -C /lib/modules/6.17.9-1-pve/build M=/var/lib/dkms/kernel-mft-dkms/4.22.1/build/mst_backward_compatibility/mst_pci CONFIG_CTF= CONFIG_CC_STACKPROTECTOR_STRONG=  modules
make[2]: Entering directory '/usr/src/linux-headers-6.17.9-1-pve'
make[3]: Entering directory '/var/lib/dkms/kernel-mft-dkms/4.22.1/build/mst_backward_compatibility/mst_pci'
  CC [M]  ../../nnt_driver/nnt_device.o
  CC [M]  ../../nnt_driver/nnt_dma.o
  CC [M]  ../../nnt_driver/nnt_pci_conf_access.o
  CC [M]  ../../nnt_driver/nnt_pci_conf_access_no_vsec.o
  CC [M]  ../../nnt_driver/nnt_memory_access.o
  CC [M]  ../../nnt_driver/nnt_ioctl.o
  CC [M]  mst_pci_bc.o
mst_pci_bc.c:6:10: fatal error: nnt_ioctl.h: No such file or directory
    6 | #include "nnt_ioctl.h"
      |          ^~~~~~~~~~~~~
compilation terminated.
make[5]: *** [/usr/src/linux-headers-6.17.9-1-pve/scripts/Makefile.build:287: mst_pci_bc.o] Error 1
make[5]: *** Waiting for unfinished jobs....
../../nnt_driver/nnt_pci_conf_access_no_vsec.c:6:5: warning: no previous prototype for ‘read_no_vsec’ [-Wmissing-prototypes]
    6 | int read_no_vsec(struct nnt_device* nnt_device, unsigned int offset,
      |     ^~~~~~~~~~~~
../../nnt_driver/nnt_memory_access.c:7:5: warning: no previous prototype for ‘write_memory’ [-Wmissing-prototypes]
    7 | int write_memory(struct nnt_device* nnt_device, struct nnt_rw_operation* write_operation)
      |     ^~~~~~~~~~~~
../../nnt_driver/nnt_dma.c:9:5: warning: no previous prototype for ‘dma_mapping_page’ [-Wmissing-prototypes]
    9 | int dma_mapping_page(unsigned int total_pinned, unsigned int page_mapped_counter,
      |     ^~~~~~~~~~~~~~~~
../../nnt_driver/nnt_memory_access.c:21:5: warning: no previous prototype for ‘read_memory’ [-Wmissing-prototypes]
   21 | int read_memory(struct nnt_device* nnt_device, struct nnt_rw_operation* read_operation)
      |     ^~~~~~~~~~~
../../nnt_driver/nnt_memory_access.c:34:5: warning: no previous prototype for ‘init_memory’ [-Wmissing-prototypes]
   34 | int init_memory(void* user_buffer, struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~
../../nnt_driver/nnt_dma.c:38:5: warning: no previous prototype for ‘pin_user_pages_in_kernel_space’ [-Wmissing-prototypes]
   38 | int pin_user_pages_in_kernel_space(unsigned int total_pages, unsigned int total_pinned,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:9:5: warning: no previous prototype for ‘clear_vsec_semaphore’ [-Wmissing-prototypes]
    9 | int clear_vsec_semaphore(struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_dma.c:77:5: warning: no previous prototype for ‘pin_user_memory_in_kernel_space’ [-Wmissing-prototypes]
   77 | int pin_user_memory_in_kernel_space(unsigned int total_pages, struct nnt_device* nnt_device,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_dma.c:123:5: warning: no previous prototype for ‘map_dma_pages’ [-Wmissing-prototypes]
  123 | int map_dma_pages(struct nnt_page_info* page_info, struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~
../../nnt_driver/nnt_dma.c:164:5: warning: no previous prototype for ‘release_dma_pages’ [-Wmissing-prototypes]
  164 | int release_dma_pages(struct nnt_page_info* page_info, struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_memory_access.c: In function ‘init_memory’:
../../nnt_driver/nnt_memory_access.c:41:59: warning: ordered comparison of pointer with integer zero [-Wextra]
   41 |     if (nnt_device->memory_device.hardware_memory_address <= 0) {
      |                                                           ^~
../../nnt_driver/nnt_pci_conf_access.c:22:5: warning: no previous prototype for ‘get_semaphore_ticket’ [-Wmissing-prototypes]
   22 | int get_semaphore_ticket(struct nnt_device* nnt_device, unsigned int* lock_value,
      |     ^~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_device.c:117:5: warning: no previous prototype for ‘create_file_name_mstflint’ [-Wmissing-prototypes]
  117 | int create_file_name_mstflint(struct pci_dev* pci_device, struct nnt_device* nnt_dev,
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access_no_vsec.c:49:5: warning: no previous prototype for ‘write_no_vsec’ [-Wmissing-prototypes]
   49 | int write_no_vsec(struct nnt_device* nnt_device, unsigned int offset,
      |     ^~~~~~~~~~~~~
../../nnt_driver/nnt_device.c:133:5: warning: no previous prototype for ‘create_file_name_mft’ [-Wmissing-prototypes]
  133 | int create_file_name_mft(struct pci_dev* pci_device, struct nnt_device* nnt_dev,
      |     ^~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:52:5: warning: no previous prototype for ‘lock_vsec_semaphore’ [-Wmissing-prototypes]
   52 | int lock_vsec_semaphore(struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_device.c:148:5: warning: no previous prototype for ‘nnt_device_structure_init’ [-Wmissing-prototypes]
  148 | int nnt_device_structure_init(struct nnt_device** nnt_device)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:110:5: warning: no previous prototype for ‘wait_on_flag’ [-Wmissing-prototypes]
  110 | int wait_on_flag(struct nnt_device* nnt_device, u8 expected_val)
      |     ^~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access_no_vsec.c:102:5: warning: no previous prototype for ‘is_wo_gw’ [-Wmissing-prototypes]
  102 | int is_wo_gw(struct nnt_device* nnt_device)
      |     ^~~~~~~~
../../nnt_driver/nnt_device.c:167:5: warning: no previous prototype for ‘create_nnt_device’ [-Wmissing-prototypes]
  167 | int create_nnt_device(struct pci_dev* pci_device, enum nnt_device_type device_type,
      |     ^~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:134:5: warning: no previous prototype for ‘set_address_space’ [-Wmissing-prototypes]
  134 | int set_address_space(struct nnt_device* nnt_device, unsigned int address_space)
      |     ^~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_device.c:231:5: warning: no previous prototype for ‘create_device_file’ [-Wmissing-prototypes]
  231 | int create_device_file(struct nnt_device* current_nnt_device, dev_t device_number,
      |     ^~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_device.c:279:5: warning: no previous prototype for ‘check_if_vsec_supported’ [-Wmissing-prototypes]
  279 | int check_if_vsec_supported(struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:204:5: warning: no previous prototype for ‘set_rw_address’ [-Wmissing-prototypes]
  204 | int set_rw_address(unsigned int* offset, unsigned int rw)
      |     ^~~~~~~~~~~~~~
../../nnt_driver/nnt_device.c:298:5: warning: no previous prototype for ‘create_devices’ [-Wmissing-prototypes]
  298 | int create_devices(dev_t device_number, struct file_operations* fop,
      |     ^~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:224:5: warning: no previous prototype for ‘read’ [-Wmissing-prototypes]
  224 | int read(struct nnt_device* nnt_device, unsigned int offset,
      |     ^~~~
../../nnt_driver/nnt_pci_conf_access.c:280:5: warning: no previous prototype for ‘write’ [-Wmissing-prototypes]
  280 | int write(struct nnt_device* nnt_device, unsigned int offset,
      |     ^~~~~
../../nnt_driver/nnt_pci_conf_access.c:334:5: warning: no previous prototype for ‘address_space_to_capability’ [-Wmissing-prototypes]
  334 | int address_space_to_capability(u_int16_t address_space)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:361:5: warning: no previous prototype for ‘get_space_support_status’ [-Wmissing-prototypes]
  361 | int get_space_support_status(struct nnt_device* nnt_device, u_int16_t address_space)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:376:5: warning: no previous prototype for ‘init_vsec_capability_mask’ [-Wmissing-prototypes]
  376 | int init_vsec_capability_mask(struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_pci_conf_access.c:403:6: warning: no previous prototype for ‘check_vsec_minimum_support’ [-Wmissing-prototypes]
  403 | void check_vsec_minimum_support(struct nnt_device* nnt_device)
      |      ^~~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_ioctl.c:14:5: warning: no previous prototype for ‘dma_pages_ioctl’ [-Wmissing-prototypes]
   14 | int dma_pages_ioctl(unsigned int command, void* user_buffer,
      |     ^~~~~~~~~~~~~~~
../../nnt_driver/nnt_ioctl.c:50:5: warning: no previous prototype for ‘read_dword_ioctl’ [-Wmissing-prototypes]
   50 | int read_dword_ioctl(void* user_buffer, struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~
../../nnt_driver/nnt_ioctl.c:80:5: warning: no previous prototype for ‘get_nnt_device_parameters’ [-Wmissing-prototypes]
   80 | int get_nnt_device_parameters(struct nnt_device_parameters* nnt_parameters, struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~
../../nnt_driver/nnt_ioctl.c:109:5: warning: no previous prototype for ‘pci_connectx_wa’ [-Wmissing-prototypes]
  109 | int pci_connectx_wa(struct nnt_connectx_wa* connectx_wa, struct nnt_device* nnt_device)
      |     ^~~~~~~~~~~~~~~
../../nnt_driver/nnt_ioctl.c:142:5: warning: no previous prototype for ‘vpd_read’ [-Wmissing-prototypes]
  142 | int vpd_read(struct nnt_vpd* vpd, struct nnt_device* nnt_device)
      |     ^~~~~~~~
../../nnt_driver/nnt_ioctl.c:187:5: warning: no previous prototype for ‘vpd_write’ [-Wmissing-prototypes]
  187 | int vpd_write(struct nnt_vpd* vpd, struct nnt_device* nnt_device)
      |     ^~~~~~~~~
make[4]: *** [/usr/src/linux-headers-6.17.9-1-pve/Makefile:2016: .] Error 2
make[3]: *** [/usr/src/linux-headers-6.17.9-1-pve/Makefile:248: __sub-make] Error 2
make[3]: Leaving directory '/var/lib/dkms/kernel-mft-dkms/4.22.1/build/mst_backward_compatibility/mst_pci'
make[2]: *** [Makefile:248: __sub-make] Error 2
make[2]: Leaving directory '/usr/src/linux-headers-6.17.9-1-pve'
make[1]: *** [Makefile:31: all] Error 2
make[1]: Leaving directory '/var/lib/dkms/kernel-mft-dkms/4.22.1/build/mst_backward_compatibility/mst_pci'
make: *** [Makefile:2: all] Error 2

# exit code: 2
# elapsed time: 00:00:04
----------------------------------------------------------------

I tried adding the blacklist config, but it didn't appear to change anything. I compared configs of the working server to the new server, and I tried copying the lines identifying my ports in /etc/network/interfaces:

iface enp65s0 inet manual
    mtu 9000

iface enp65s0d1 inet manual
    mtu 9000

Rebooted the server and they're now listed in the GUI, but even after adding a bridge with an IP I'm not able to ping out on that subnet. Well, it was worth a shot!

 

[alexskysilk]

Since mst 4.22 seems to be broken on 6.17, you have a few alternatives:

1. install an older kernel. 6.14 and 6.11 are both available. 6.8 is too I think.
2. use a newer version of mst. The most current version is 4.34.1-10
3. boot from a thumbdrive with a linux livecd.

Once you set the operating mode, it will remain that way permanently and you can resume normal operation. Once that is done, you can identify the device id by searching "dmesg | grep eth" and "ip a"

use the resulting device IDs to form your interfaces entries.

 

[Vampire337]

Since mst 4.22 seems to be broken on 6.17, you have a few alternatives:

1. install an older kernel. 6.14 and 6.11 are both available. 6.8 is too I think.
2. use a newer version of mst. The most current version is 4.34.1-10
3. boot from a thumbdrive with a linux livecd.

Once you set the operating mode, it will remain that way permanently and you can resume normal operation. Once that is done, you can identify the device id by searching "dmesg | grep eth" and "ip a"

use the resulting device IDs to form your interfaces entries.

1. Trying to list & download previous kernels in my PVE 9.1, I don't see an option for anything but 6.17:

# apt search pve-kernel
pve-firmware/stable,now 3.17-2 all [installed]
  Binary firmware code for the pve-kernel

# proxmox-boot-tool kernel list
Manually selected kernels:
None.

Automatically selected kernels:
6.17.2-1-pve
6.17.9-1-pve

Additionally, https://pve.proxmox.com/wiki/Proxmox_VE_Kernel#Proxmox_VE_9 indicates PVE 9.0 can use 6.14, while 9.1 (which I have installed) can only use 6.17. Is this consistent with what I'm seeing here?

2. I'm not opposed to using a newer version of MST, but that linked forum post suggests that ConnectX-3 may not be supported by anything newer than 4.22.

3. I think I'll start from here, but rather than trying to guess what version of Linux livecd supports MST, I'll just install PVE 8 on the server and run MST from there. Now that I know the changes apply to the card itself, should be no problem to get it set the way I have the other one, then install 9.1 fresh and maybe it just appears like the onboard NICs?

Thanks again for the help!

 

[alexskysilk]

1. try proxmox-kernel
2. says who? while the DRIVERS arent provided in the package, mst should work just fine. if it doesnt, you can always uninstall
3. that is one way; another would be to make a persistent live usb so you dont have to mess with your install at all.