Question about cluster and performances

S

spamlabfab

New Member
Jan 30, 2025
1
0
1
Hello,

I've got a quite basic setup:
3 HPEs DL380 servers with 5 nvme disks and a 10 GbE card.
I'm building a cluster of those 3 servers with CEPH storage.
the 10 GbE connectivity is conformed ok through iperf3
Ceph is set up on the 10 GbE network, rados bench sound ok. In a first look, I'd really say connectivity doesnt look like a problem.

though, when im doing VMs on the ceph storage, benches show that read/write dont go much more above 220 MB/s which should be way more.
is there some things to be done to reach full disk performance?
thanks a lot

Fab
 
spamlabfab said:
Hello,

I've got a quite basic setup:
3 HPEs DL380 servers with 5 nvme disks and a 10 GbE card.
I'm building a cluster of those 3 servers with CEPH storage.
the 10 GbE connectivity is conformed ok through iperf3
Ceph is set up on the 10 GbE network, rados bench sound ok. In a first look, I'd really say connectivity doesnt look like a problem.

though, when im doing VMs on the ceph storage, benches show that read/write dont go much more above 220 MB/s which should be way more.
is there some things to be done to reach full disk performance?
Click to expand...

The consensus in earlier discussions was that 10Gbe is the absolute minimum ( 25 GBe is better, 100 GBe recomnended for new setups by independent consultants in this community) and enterprise ssds with power-loss-protections are the way to go. Due to the latency requirements of cluster communication you should also have a dedicated network for corosync ( 1Gbe should be enough IIRC).

So you need at least three neworks:
- Ceph
- Corosync/Cluster communication
- Outside World ( aka clients accessing the vms Services and communication between guests)

See also @UdoB exzellent writeup on small Clusters:
UdoB

[TUTORIAL] Thread 'FabU: can I use Ceph in a _very_ small cluster?'

Ceph is great, but it needs some resources above the theoretical minimum to work reliably. My assumptions for the following text:
  • you want to use Ceph because... why not?
  • you want to use High Availability - which requires Shared Storage (note that a complete solution needs more things like a redundant network stack and power supplies)
  • you want to start as small (and cheap) as possible, because this is... “only” a Homelab
You plan for three Nodes. Each node has s single dedicated disk for use as an “OSD”. This is the documented minimum...
  • Like

The PVE wiki pages on HCI and Clusters are also worth a read:
https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

https://pve.proxmox.com/wiki/Cluster_Manager

In the second page you should at least read the sections on cluster Networks and redundant links
 
Last edited:
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom