Dumb CEPH questions

tcabernoch

tcabernoch

Active Member
Proxmox Subscriber
Apr 27, 2024
484
178
43
Portland, OR
www.gnetsys.net
Hello. Some of you know me. Feel free to tell me how dumb I am, please.

I need to setup DR replication between two geographically remote CEPH clusters.
If I can deliver on this, we are gonna have good DR, like Zerto used to give us on VMware.
And delivering isn't really optional. It absolutely has to work. We are gonna sell it. Probly already have.
Got the first cluster built. Starting number two ...

I need to make a CEPH cluster at a site with the gear on hand.
I have budget for disks, but I'm stuck with these hosts.
I would have to justify complete disk replacements tho.

I have 5 x Gen 14/15 Dell PowerEdge servers.
Three of them have 8 bays. No other OS boot option, so really only 7 SSD to contribute to CEPH.
Two of them have 24 bays. Full. Lots of disks.
All disks are 960gb SSD.

I know I'm gonna lose a lot of capacity.
My first cluster of 4 hosts w 7 x 960gb SSD provides 8TB (in theory) of usable space
From my limited experience with CEPH, I probably shouldn't count on more than 9TB out of this array that I'm planning.
And that really sux. Hard to do business without sufficient storage.

So, here's the dumb part.
Two of these hosts have LOTS of extra 960gb SSD.
I know its not recommended. I know its not optimal.
But what would happen if I added two extra OSD from each of these well-provisioned hosts to the CEPH array?

Right, "ur not sposed to do that".
But if I did, what should I expect?
 
Last edited:
What’s your network speed between clusters?

Re unbalanced storage, if a “big” node dies Ceph will need to replicate 24 OSDs to the remaining nodes. Perhaps it will be less per OSD but the point is Ceph will try to remain level, for lack of a better term.
 
  • Like
Reactions: tcabernoch
tcabernoch said:
I was thinking about 10 OSDs contributed by the 'big' hosts, with only 7 contributed by the others.
I'm still not at all sold on the idea. Very ready to hear that its way too dumb.
Click to expand...
its way too dumb ;) since your crush rule requires an osd on three different nodes, and you have two nodes with excess capacity... the excess capacity is unused.

but I think you're going about this the wrong way. What is your REQUIRED usable capacity? might need to replace one server. or get bigger drives.
 
  • Like
Reactions: tcabernoch
Thanks Alex.

The requirement is site to site replication. CEPH will be the tool. The rest will sort out as needs be.

"crush rule requires an osd on three different nodes, and you have two nodes with excess capacity... the excess capacity is unused"
Ok. I don't have deep understanding of crush yet. I've looked at it, read about it, haven't watched it really in action yet.
I'll do some more reading, and try to validate what you are saying here.

===========================
EDIT ... Um ... One of the 7 disk host has a pair of NVMe that I could use for boot.
3 of the hosts could have 8 disks.

host1 - (1 OS disk) + 7 OSD SSD
host2 - (1 OS disk) + 7 OSD SSD
host3 - (raid1 nvme OS disk) + 8 OSD SSD
host4 - (whatever) + 8 OSD SSD
host5 - (whatever) + 8 OSD SSD

I still know next to nothing about crush rules. That 2x7disk/3x8disk plan might be just as dumb.
 
Last edited:
You must log in or register to reply here.
Top Bottom