[SOLVED] Infiniband Device to LXC

[elurex]

How to passthrough an infiniband device to LXC?

according to

• https://lxd.readthedocs.io/en/stable-3.0/containers/#device-types
• https://github.com/lxc/lxd/issues/3983
• https://docs.mellanox.com/pages/rel...D)overInfiniBandfabric.-PreparingLXCContainer

It all can be done with lxd or lxc. But how to do it on PVE 6.1?

it needs more that just ib0 network functionality passthrough

lxc.net.1.type: phys
lxc.net.1.flags: up
lxc.net.1.link: ibp36s0d1v0
lxc.net.1.name: ib0
lxc.net.1.ipv4.address: 10.0.1.100/24

But it also needs

• One issmX char device per function and per port
• One umadX char device per function and per port
• One uverbsX char device per function

in /dev/infiniband

 

[elurex]

Currently I am following the reference from

• https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html#lbAT
• https://blog.blaok.me/pass-a-device-to-a-linux-container/

 

[elurex]

I solve my own problem

For LXC.conf adds the following

lxc.cgroup.devices.allow: a 231:* rwm
lxc.cgroup.devices.allow: a 10:54 rwm
lxc.cgroup.devices.allow: a 10:229 rwm
lxc.autodev: 1
lxc.hook.autodev: /var/lib/lxc/302/mount-hook.sh
lxc.apparmor.profile: unconfined
lxc.mount.auto: cgroup:rw
lxc.mount.auto: proc:rw
lxc.mount.auto: sys:rw
lxc.net.1.type: phys
lxc.net.1.flags: up
lxc.net.1.link: ibp36s0d1v0
lxc.net.1.name: ib0
lxc.net.1.ipv4.address: 10.0.1.102/24

for mount-hook.sh

#!/bin/sh
cp -dpR /dev/infiniband* ${LXC_ROOTFS_MOUNT}/dev
mknod -m 766 ${LXC_ROOTFS_MOUNT}/dev/fuse c 10 229

 

[alpha754293]

Sorry for resurrecting such an old thread:

I applied these instructions to my 100 Gbps IB deployment and when I check ip a, this is what I get:

8: ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc mq state DOWN group default qlen 256
    link/infiniband 00:00:04:a6:fe:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

So, my question is how did you bring the interface up?

From the changes that you've made to your <<CTID>>.conf file, it looks like that you're using virtual functions as well.

How did you get your iB switch to recognise that there are virtual functions running on the same physical link cable (DAC or (active) optical cable)?

Did you have to do anything special either on your client or on your switch end?

Your help is greatly appreciated.

Thank you.