LXC SRIOV passthrough shows no-carrier

[ewaldo]

I have a Mellanox ConnectX-3 card configured with VFs connected to Voltaire 4036's that are running my subnet manager. I have ib_ipoib loaded on the Proxmox host and can successfully assign an IP to the VF and ping my other IB hosts. When I pass the VF to a [un]priviliged LXC container with or without apparmor disabled, ip link shows the interface with NO-CARRIER and does not work. I've tried setting a type of infiniband, but the current LXC version in Proxmox does not recognize this type.

lxc.net.1.type: phys
lxc.net.1.flags: up
lxc.net.1.link: ibp8s0v2
lxc.net.1.ipv4.address: 192.168.4.22/24
lxc.net.1.mtu: 65520

root@local:~# ip link | grep ibp8s0v2
11: ibp8s0v2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 65520 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 256

root@local:~# cat /sys/class/net/ibp8s0v2/mode
connected

 

[ewaldo]

Looks like I'll be answering my own question. The problem is I was running my subnet managers on the Voltaire switches.

https://docs.mellanox.com/pages/vie...zation(SR-IOV)-VirtualFunctionInfiniBandPorts has the subtle note "The SM on QDR switches do not support SR-IOV."

Switched to running opensm on my proxmox hosts and the no-carrier problem goes away and everything is functional.