Debian kernel for OFED stack

[fomichevk]

Hello.
I need to use OFED stack on Proxmox for accessing data storage via Infiniband.
Unfortunately, latest OFED stack built only for Debian stock kernel. Module assistant fails when building ofa-kernel on pve-kernel and pve-headers, but everything works perfectly on linux-image and linux-headers.
If I use Debian kernel in Proxmox in production, what I miss? Only OpenVZ support or something else?
Thanks in advance.

Upd:
Well, i've search forum and found this thread: http://forum.proxmox.com/threads/4868-What-the-differences-between-Debian-kernel-and-pve-kernel
Okay, there is many differences, but how can I solve my problem without changing kernel?
Maybe, you have plans to support Infiniband out-of-box?

 

[e100]

I've tried a million ways to get OFED to compile too, no luck, gave up.

I am using IPoIB with no problem, that is included in the Linux Kernel.
Wanted to also use SDP, but with no OFED IPoIB is all I can get working.

There are some other Infiniband modules included in the Linux kernel without OFED, maybe it already has what you need:
ib_iser.ko
ib_srp.ko

 

[e100]

If I use Debian kernel in Proxmox in production, what I miss? Only OpenVZ support or something else?

This is a good question, can we use the stock debian kernel with proxmox if we only want to use KVM?

 

[dietmar]

This is a good question, can we use the stock debian kernel with proxmox if we only want to use KVM?

We do not use that kernel, so we can't answer that question (you will end up with an untested system).

 

[e100]

I have been looking into compiling OFED again and think I may be close to a solution.
Maybe between the two of us we can figure this out.

I downloaded OFED-1.5.4.tgz
Then I installed a few items:
apt-get install alien rpm zlib1g-dev pve-headers-2.6.32-6-pve quilt

extracted ofed and ran the install.pl
All went well until the kernel modules

after if barfed I went to the build directory
/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4

did make clean
edit config.mk:
BACKPORT_INCLUDES=-I${CWD}/kernel_addons/backport/2.6.32-EL6.1/include/

look at patch:
kernel_patches/backport/2.6.32-EL6.1/uverbs_0010_to_2_6_32_el6.patch

apply patch to:
/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4/drivers/infiniband/core/uverbs_main.c

ran make and it completed without error.

I think the problem with the OFED distribution is that it thinks this is a generic 2.6.32 kernel but it is much closer to a 2.6.32-EL6.1 kernel.
So if we can get OFED to think this is EL6.1 we should be able to simply use the install.pl to get everything compiled.

I think we only need to patch:
ofed_scripts/get_backport_dir.sh

Need a test like this:
2.6.32-6-pve*)
echo 2.6.32-EL6.1
;;

 

[term]

Hi,

Have you had any additional success, e100? Many thanks for your IB writeup in the wiki!

Has anyone been able to turn a ib interface into a bridge that can be used by proxmox vms? I've tried, but have so far been unsuccessful.

 

[e100]

Hi,

Have you had any additional success, e100? Many thanks for your IB writeup in the wiki!

Has anyone been able to turn a ib interface into a bridge that can be used by proxmox vms? I've tried, but have so far been unsuccessful.

I have not tried to get OFED stuff working for a long time.
I do not think it will ever be possible to make ib bridge useable by vms.

 

[term]

I have not tried to get OFED stuff working for a long time.
I do not think it will ever be possible to make ib bridge useable by vms.

Well, I have figured out how to expose my ancient topspin HCA via pci pass through. It shows up in windows 2012, but I have not found any drivers yet. I doubt I will be able to share a card among multiple VMs.