New Ceph cluster advice

jeversole

New Member
Aug 26, 2024
4
1
3
Hello,

I'm currently going through the process of migrating VMs from VMWare to Proxmox. I have 3 nodes in the Proxmox cluster. My intention is to convert the old VMWare hosts into a Ceph cluster once everything is moved over to the new servers. There are 3 of those and I have several old servers I can add if needed. I've read that while a 3 node Ceph cluster works, a 4 node cluster is the ideal minimum. Having no experience with Ceph as of yet, I cannot validate the validity of that.

The old VM hosts are Dell Poweredge R720s using dual Xeon E-5 2667v2 @ 3.3GHz CPUs and 256GB DDR3 RAM. As I said, there are 3 of those. Out of the half dozen or so other servers that are soon to be retired, the next closest thing I have to the R720 specs is a R720xd with dual E5-2660 @ 2.2GHz CPUs and 128GB of DDR3 RAM.

My question is, will using a 4th node with that big of a difference in specs have a huge effect on performance? I have 36 x 2TB SSDs (72TB Raw capacity before parity) to distribute across 4 servers. I just want to make sure this will work efficiently. Thanks in advance!
 
  • Like
Reactions: unsichtbarre
There is enough RAM available for 9 SSD OSDs in each host. You usually calculate with 5 GB RAM per OSD.
Thanks for the intel. Will the 4th node having less processing power cause any discernable difference in performance? My understanding is that Ceph stripes across all nodes in the cluster so I assume that processing speed would default to the lowest spec.
 
I run a 5-node R720 Ceph cluster in production. I made sure to flash the PERC to IT-mode using this guide https://fohdeesha.com/docs/perc.html

Ceph is scale out and loves lots of OSDs. Not hurting for IOPS with workloads ranging from DBs to DHCP servers.

I use the following optimizations learned through trial-and-error. YMMV

Code:
    Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])

    Set VM Disk Cache to None if clustered, Writeback if standalone

    Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option

    Set VM CPU Type to 'Host'

    Set VM CPU NUMA on servers with 2 or more physical CPU sockets

    Set VM Networking VirtIO Multiqueue to 1

    Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows

    Set VM IO Scheduler to none/noop on Linux

    Set Ceph RBD pool to use 'krbd' option
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!