Proxmox cluster

[Lighterfield]

Hi,

New proxmox user here.
Planning on migrating our current vmware vsan servers to proxmox. (14 VM's)

Currently in process of ordering 3x HP DL380gen11 servers. (not ordered yet)
2x intel 6530 gold cpu/512GB ram/node, rest of the hardware seems ok except the thing that worries me is the 4X 6TB SAS disks @7200rpm.
Will this be a bottleneck when using a HA cluster -3 nodes- with ceph?

Thanks,

 

[Lukas Moravek]

Hi @Lighterfield

generally it should not be bottleneck (depends on your usage), but it is verry general question without knowing what you runs on VM.

- Clearly DB with x thousands of R/W operation is not good idea on HDD. For DB is better ZFS + Replication on SSD.
- Bottleneck can be also you network connection (allocate 1 NIC with according speed) for CEPH

L.

 

[Lighterfield]

Hi @Lighterfield

generally it should not be bottleneck (depends on your usage), but it is verry general question without knowing what you runs on VM.

- Clearly DB with x thousands of R/W operation is not good idea on HDD. For DB is better ZFS + Replication on SSD.
- Bottleneck can be also you network connection (allocate 1 NIC with according speed) for CEPH

L.

Hi Lukas, thanks for the reply.
VM's are the usual windows DC's, DHCP/print/file servers, and some client-server ubuntu db applications with small client usage. (+-20 concurrent users)
One NIC will be dedicated for Ceph (10Gbit)

 

[Lukas Moravek]

If there is no huge traffic for file servers (classics share for users data - office use), configuration is OK.

 

[MarkusKo]

if you have the budget for that hardware, dont waste your time with hard disks, use ssd's. 25G nics (LACP bond) for ceph would be a good idea.
storage/networking will be your main bottleneck here, especially with ceph.

check the official ceph benchmarks for some numbers

 

[Johannes S]

Adding to MarkusKos comment you might also want to read @UdoB great writeup on using Ceph in small custers:

[TUTORIAL] Thread 'FabU: can I use Ceph in a _very_ small cluster?'

Dec 26, 2024

Ceph is great, but it needs some resources above the theoretical minimum to work reliably. My assumptions for the following text:

• you want to use Ceph because... why not?
• you want to use High Availability - which requires Shared Storage (note that a complete solution needs more things like a redundant network stack and power supplies)
• you want to start as small (and cheap) as possible, because this is... “only” a Homelab

You plan for three Nodes. Each node has s single dedicated disk for use as an “OSD”. This is the documented minimum...

• UdoB
• ceph cluster mini pc minimize shared filesystem storage
• Replies: 22
• Forum: Proxmox VE: Installation and configuration

• 

Basically you will need fast network (10Gb/s minimum, 25Gb/s recommended ) and enterprise-ssds with powerloss-protection to get good performance.


The documentation also recommends to using dedicated networks for the storage/ceph network, the cluster (corosync) network and the client network, so you will need at least three seperate network links. Please note that the 10Gb/s/25Gb/s recommendation is only for ceph, for corosync latency is more important than performance. And for the clients you will know best which network performance your services need

https://pve.proxmox.com/wiki/Deploy...r#_recommendations_for_a_healthy_ceph_cluster
https://pve.proxmox.com/wiki/Cluster_Manager#_cluster_network

 

[Lighterfield]

Ordered 6x 3.84TB SSD's and 2x 10Gbit NICs per server. For 12 vm's this should be sufficient for now I guess. Thanks all for the replies.