Proxmox 6.1 with Ceph and RoCE

[zeuxprox]

Hi,

in a few weeks I will have to set up a 4 nodes cluster with ceph. Each node will have 4 nvme osd and I would like to use RoCE for ceph public network and ceph cluster network. My network card is Supermicro AOC-M25G-m4s, practically a Mellanox ConnectX-4LX, with 4 ports 25Gb. I would like to use 2 ports in bondig alb for ceph public network and 2 ports in bonding alb for ceph private network. My question is: how can I configure Proxmox to use RoCE for ceph? I have 2 cumulus express switch 25Gb (CX-5148-S) and I'll setup 2 VLANs, one for ceph public and one for ceph cluster.

Thank you very much

 

[wolfgang]

Hi,

Proxmox VE does not support RoCE.
XIO is not enabled in our ceph packages, which is needed for RoCE.

https://software.intel.com/en-us/ar...nologies-to-accelerate-ceph-storage-solutions

 

[zeuxprox]

Can we expect the introduction of this feature in future versions?

Thank you wolfgang.

 

[wolfgang]

As long xio is flaged as experimental in ceph the answer is no.
Also, I'm not sure if this more memory and CPU usage over the latency improvement brings a benefit for HCI setups.

https://docs.ceph.com/docs/luminous/dev/messenger/?highlight=xio