1. J

    [TUTORIAL] Proxmox 8 Mellanox Infiniband and SR-IOV

    Here is how i was able to get proxmox working with Infiniband and SR-IOV. Hardware used is a mellanox switch (sx6036) and a mellanox Cx-4 100gbps EDR dual (or single) port card. Make sure the firmware is latest. AS FAR AS IM AWARE, THIS WILL NOT WORK WITH OPENSM AND MUST HAVE A MELLANOX SWITCH...
  2. R

    CEPH in PVE 7.3 can not working with RDMA/RoCE?

    Hi, PVE geekers: I build a pve cluster on three server (with ceph), with pve & ceph package version as follow: root@node01:~# pveversion pve-manager/7.3-3/c3928077 (running kernel: 5.15.74-1-pve) root@node01:~# ceph --version ceph version 16.2.13 (b81a1d7f978c8d41cf452da7af14e190542d2ee2)...
  3. H

    CEPH traffic on Omnipath or Infiniband NICs and switches?

    Hi all, We are looking in to deploying a new refurbished NVME HCI Ceph Proxmox cluster. At this point we look at 7 nodes, each with 2 NVME OSD drives, with expansion for 2 NVME OSD's more. As we would quickly saturate a 25GbE link we should be looking in to 40/50/100 GbE links and switches...
  4. R

    help(?!) building OFED drivers

    I am trying to build OFED drivers from source for my ConnectX3 Pro cards so that I can make one port 40/56 Gb Inifiniband and the second port as 40 Gb Ethernet. This configuration requires OFED drivers as far as I understand. mlx4_core and mlx4_en wont work simultaneously that I am aware of -...
  5. S

    Netzwerk, Infiniband = Unbekannt

    Hallo Forum, ich habe das Problem, dass meine Infiniband-Karte (Mellanox X3), die ich für IPoIB nutze in der Proxmox GUI immer als "Unbekann" deklariert ist und dich die nur über die Shell bearbeiten kann. Ist dieses Problem bekannt?
  6. C

    Help setting iser transport for ZFS on ISCSI

    Hi all, I'm new to Proxmox having been running VMWare since maybe 3.5 or something. Decided to switch over because I wanted to take the cheaper path to Infiniband and then found the support in VMWare not quite there. My setup now is Proxmox server with a dual-port Mellanox ConnectX-3 card. My...
  7. DrillSgtErnst

    [TUTORIAL] ConnectX VPI & Proxmox 6.2

    Hello Fellas, Since there are many many many posts about people trying to get cheap Mellanox ConnectX VPI 40GB/s working on Proxmox and I really had a hard time getting mine to work, I am willing to share my Insights, and hope to make some peoples days. At First. I am running a cluster of 4...
  8. N

    LACP two 40 gbit/s eth or 40 gbit/s eth + 56 gbit/s infiniband

    I am upgrading a 16 node cluster that has 2 NVMe drives and 3 SATA drives used for ceph. My network cards are Mellanox MCX354A-FCBT and have 2 QSFP ports that can be configured as Infiniband or Ethernet. My question is how best should I utilize the two ports. My options are: 1) LACP into VPC...
  9. Y

    Proxmox + Mellanox 3X 40G Poor bandwidth performance

    Hello, i'm testing the performance over two nodes connected by two Mellanox: MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0) On Proxmox 6 last version i have installed all pakages: apt-get install rdma-core libibverbs1 librdmacm1 libibmad5 libibumad3 librdmacm1 ibverbs-providers...
  10. E

    LXC SRIOV passthrough shows no-carrier

    I have a Mellanox ConnectX-3 card configured with VFs connected to Voltaire 4036's that are running my subnet manager. I have ib_ipoib loaded on the Proxmox host and can successfully assign an IP to the VF and ping my other IB hosts. When I pass the VF to a [un]priviliged LXC container with or...
  11. H

    Mellanox Need help setting up Infiniband

    Hi, I have 2x ConnectX-2 Cards and need to setup infiniband, RDMA over NFS on Proxmox 5.4. I followed a few guides online but I'm unable to change protocol type from ETH to IB. What is correct way of setting the port type on Proxmox?
  12. Y

    [SOLVED] Infiniband Mellanox 40Gb Speed Low

    Hi, first, merry christmas guys! I just installed 2 mellanox infiniband cards (40Gb Connectx-2) each on on server and installed after that the latest mellanox drivers from here: Finally i made a test with iperf but i get only this...
  13. C

    Infiniband, ZFS, NFS?

    Hello all, I'm hoping you maybe able to help me as I seem a bit in over my head this time. I have three physical servers that I am working to build into an HA cluster. Two of those servers will be the processing power (hereafter referred to as P1 and P2) of the cluster and one is the storage...
  14. J

    Running a VM from a Glusterfs filesystem

    I have a Glusterfs system running over Infiniband which I'm testing with Proxmox 3.4 (soon to be upgraded to 4.4). The vzdumps work great but I'm wondering how I can run my VM images directly from the Glusterfs mount point without a huge performance degradation. I tried SATA disks with obvious...
  15. G

    OFED for PVE

    Greetings, This is my first post so please be gentle. Has anybody had success in installing OFED for Debian Jessie? I tried the install script from Mellanox but it seems to be hard-coded to Debian 8.2 or 8.1. PVE 4.1 is on Debian 8.3, which is what I was running when I spent two days trying...
  16. A

    SCST / ib_srpt / isert-scst in releases ?

    Hello, is there any change to get SCST together with srpt and the required kernel-modules in the releases ? Current scst compiles fine against Kernel 4.4.6 and in-tree ofed / rdma packages. Having those packages / kernel modules in the release would skip the need to compile / build with each...
  17. I

    Proxmox VE 4.1 Infiniband problem

    Hi. I am sorry for my bad english :( I have 2 cluster on the Proxmox VE ver. 3.4 and 4.1 All equipment is absolutely identical in both clusters All nodes has 2*X5650 Xeon, 96Gb RAM and InfiniBand Card: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT / s - IB QDR / 10GigE] (rev b0) Both...


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!