Virtual Connectx-3 networked VMs project

T

top_east

New Member
Aug 21, 2025
3
0
1
I am trying to homelab some MPI issues and was hoping to be able to use a connectx-3 card, SR-IOV and some VFs on that card, passed through to Ubuntu VMs for that purpose.

Note that I need for these cards to be in Infiniband mode (not ethernet) and almost all the guides I found for working with these cards (like https://blog.joeplaa.com/mellanox-connectx-3-on-proxmox-8-2/) were for ethernet setups as a cheap way to get 10G networking.

I followed that guide and I do have my connectx-3 card recognized, with 8 VFs:


Bash: 
root@proxmox:~# lspci | grep Mel
01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
01:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
01:01.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]

Also the NICs show up:

Bash: 
root@proxmox:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 6c:4b:90:18:fa:2a brd ff:ff:ff:ff:ff:ff
3: ibp1s0: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:08:fe:80:00:00:00:00:00:00:24:8a:07:03:00:fa:be:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
Truncated VF list: ibp1s0
5: ibp1s0v1: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:0a:fe:80:00:00:00:00:00:00:62:ec:e4:d0:3f:ec:95:4b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: ibp1s0v2: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:0b:fe:80:00:00:00:00:00:00:12:a3:b6:5a:b5:e5:f7:06 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
7: ibp1s0v3: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:0c:fe:80:00:00:00:00:00:00:1a:22:de:e3:91:14:e1:09 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
8: ibp1s0v4: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:0d:fe:80:00:00:00:00:00:00:ea:a9:1d:6e:66:21:1f:c5 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
9: ibp1s0v5: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:0e:fe:80:00:00:00:00:00:00:fe:95:79:4e:f0:1d:26:77 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
10: ibp1s0v6: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN mode DEFAULT group default qlen 256
    link/infiniband 80:00:0a:0f:fe:80:00:00:00:00:00:00:02:8e:7a:df:33:c8:e0:25 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
11: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 6c:4b:90:18:fa:2a brd ff:ff:ff:ff:ff:ff
12: tap105i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr105i0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 8a:7f:2b:54:2e:6b brd ff:ff:ff:ff:ff:ff
13: fwbr105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ee:40:20:37:af:60 brd ff:ff:ff:ff:ff:ff
14: fwpr105p0@fwln105i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether c6:82:a2:c9:e2:f5 brd ff:ff:ff:ff:ff:ff
15: fwln105i0@fwpr105p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr105i0 state UP mode DEFAULT group default qlen 1000
    link/ether ee:40:20:37:af:60 brd ff:ff:ff:ff:ff:ff

The interfaces do not show up in the Proxmox gui, however, but that might be because they are infiniband rather than ethernet (like in the guide I followed)

I have passed through two of the VFs to a pair of VMs and am running opensm on one of them, but I don't see any communication between VFs.

I had read somewhere that these VFs would be able to see each other without connecting the card to a switch, but I'm not sure that's correct. If I have to get a switch that would be unfortunate as they are huge and possibly noisy.

Bash: 
root@ompi-node01:~# service opensm status
● opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers
     Loaded: loaded (/usr/lib/systemd/system/opensm.service; enabled; preset: enabled)
     Active: active (exited) since Thu 2025-08-21 22:15:03 UTC; 1min 23s ago
       Docs: man:opensm(8)
    Process: 685 ExecCondition=/bin/sh -c if test "$PORTS" = NONE; then echo "opensm is disabled via PORTS=NONE."; exit 1; fi (code=exited, s>
    Process: 697 ExecStart=/bin/sh -c if test "$PORTS" = ALL; then PORTS=$(/usr/sbin/ibstat -p); if test -z "$PORTS"; then echo "No InfiniBan>
   Main PID: 697 (code=exited, status=0/SUCCESS)
        CPU: 17ms

Aug 21 22:15:03 ompi-node01 systemd[1]: Starting opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers...
Aug 21 22:15:03 ompi-node01 sh[697]: Starting opensm on following ports: 0x86cb9e4358e836bf
Aug 21 22:15:03 ompi-node01 systemd[1]: Finished opensm.service - Starts the OpenSM InfiniBand fabric Subnet Managers.
root@ompi-node01:~# ibnetdiscover 
ibwarn: [1009] _do_madrpc: send failed; Invalid argument
ibwarn: [1009] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0)
./libibnetdisc/ibnetdisc.c:807; Failed to resolve self
ibnetdiscover: iberror: failed: discover failed

root@ompi-node01:~# ibstat
CA 'ibp0s16'
    CA type: MT4100
    Number of ports: 1
    Firmware version: 2.40.5048
    Hardware version: 1
    Node GUID: 0x00140500801165d7
    System image GUID: 0x248a070300fabed3
    Port 1:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x86cb9e4358e836bf
        Link layer: InfiniBand
root@ompi-node01:~# ibnetdiscover 
ibwarn: [1011] _do_madrpc: send failed; Invalid argument
ibwarn: [1011] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0)
./libibnetdisc/ibnetdisc.c:807; Failed to resolve self
ibnetdiscover: iberror: failed: discover failed

root@ompi-node02:~# ibstat
CA 'ibp0s16'
    CA type: MT4100
    Number of ports: 1
    Firmware version: 2.40.5048
    Hardware version: 1
    Node GUID: 0x001405001d16539a
    System image GUID: 0x248a070300fabed3
    Port 1:
        State: Down
        Physical state: Polling
        Rate: 10
        Base lid: 0
        LMC: 0
        SM lid: 0
        Capability mask: 0x02514868
        Port GUID: 0x6a27859f0c053781
        Link layer: InfiniBand

Am I missing something here? Do I need to add the parent interface to a bridge in proxmox or something?
 
are you doing all this on a single machine? IB loopbacks are tricky. easiest way is with a two port card and a cable between both ports. run the subnet manager on the physical not virtual function and make sure to enable virt support in opensm.conf
 
  • Like
Reactions: top_east
Yes I'm doing it on a single machine, unfortunately I got a single port card, I guess I can get a 2-port card and try again, since it seems that I'm on the right track. At least the card was only $15.

Thanks!
 
You must log in or register to reply here.
Top Bottom