We are using PVE and Ceph in Dell blades using the M1000e modular chassis.
We are currently using dual mezzanine cards with 2x 10 GbE ports, one for Ceph front-end and one for Ceph back-end.
Public LANs, guest LANs, and Corosync are handled by 4x 10GbE cards on 40GbE MXL switches, so all is good there.
OSDs are 8TB intel P4510 NVMe/PCIe 3.0 drives, and I need to know the best way to build the ceph networks to get the most out of them.
As the Ceph front-end and back-end are each composed of 20 Gb LAGs, our i/o bottleneck is quite obvious and our benchmark testing pegs right to 2 GB/s. If I combine the ceph nets onto a single 40 Gb LAG, performance is slightly worse.
I just need to know, given the constraints of the M1000e, what is the best way to build the Ceph front/back end networks for the best guest VM performance? What is the general community experience and sentiment?
There are 56 Gb and 40 Gb infiniband options for our hardware that we are looking at to try and move the bottleneck as far out as we can.
Is infiniband supported on PVE/Ceph? Do you run them in pure infiniband mode, or ethernet mode, or IPoIB? Is RDMA working/do you use it? What kind of performance are you getting?
All the available NIC options are based on ConnectX-3 silicon. The FDR switch is the M4001F and the FDR10 switch is the M4001T, so apparently the FDR and FDR10 parts are not interchangeable. I am shooting for at least 56 Gb anyway.
All the NICs are dual-port NICs and there will be 4 switches in the chassis. How can I combine / aggregate / bond the links to effectively get 112 Gb?
We are currently using dual mezzanine cards with 2x 10 GbE ports, one for Ceph front-end and one for Ceph back-end.
Public LANs, guest LANs, and Corosync are handled by 4x 10GbE cards on 40GbE MXL switches, so all is good there.
OSDs are 8TB intel P4510 NVMe/PCIe 3.0 drives, and I need to know the best way to build the ceph networks to get the most out of them.
As the Ceph front-end and back-end are each composed of 20 Gb LAGs, our i/o bottleneck is quite obvious and our benchmark testing pegs right to 2 GB/s. If I combine the ceph nets onto a single 40 Gb LAG, performance is slightly worse.
I just need to know, given the constraints of the M1000e, what is the best way to build the Ceph front/back end networks for the best guest VM performance? What is the general community experience and sentiment?
There are 56 Gb and 40 Gb infiniband options for our hardware that we are looking at to try and move the bottleneck as far out as we can.
Is infiniband supported on PVE/Ceph? Do you run them in pure infiniband mode, or ethernet mode, or IPoIB? Is RDMA working/do you use it? What kind of performance are you getting?
All the available NIC options are based on ConnectX-3 silicon. The FDR switch is the M4001F and the FDR10 switch is the M4001T, so apparently the FDR and FDR10 parts are not interchangeable. I am shooting for at least 56 Gb anyway.
All the NICs are dual-port NICs and there will be 4 switches in the chassis. How can I combine / aggregate / bond the links to effectively get 112 Gb?
- i'm sure I can fine tune that but it's enough for my needs.