Adding a node to a cluster with Ceph storage

[trilljester]

Hi there,

We have a cluster that has 4 nodes with Ceph storage. One of those nodes died yesterday, which puts the cluster in danger of freezing if another node goes down. I'm working on getting the dead node repaired and back online, but it will take a few weeks to get parts for it and get it fixed and running again.

In the meantime, could I add a server that does NOT have any storage on it just to keep the cluster going in case another node goes down? I have a couple of servers that are loaded in RAM and CPU but do not have any storage. If I could add one of those in there just to be able to run VM's, would that work?

And how would I go about adding that node to the cluster?

 

[UdoB]

which puts the cluster in danger of freezing if another node goes down.

Are you talking about PVE Quorum? Just add a Quorum device: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support to get to an odd number of nodes to vote - as recommended for even numbers of nodes.

Are you talking about Ceph? Verify that it got re-balanced to store a copy of everything on each of the three survivors. (If you use the default size=3/min=size=2.)

And yes, of course you can add just another full blown server, with or without actual storage. But a QDevice should be sufficient.

 

[trilljester]

Are you talking about PVE Quorum? Just add a Quorum device: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support to get to an odd number of nodes to vote - as recommended for even numbers of nodes.

Are you talking about Ceph? Verify that it got re-balanced to store a copy of everything on each of the three survivors. (If you use the default size=3/min=size=2.)

And yes, of course you can add just another full blown server, with or without actual storage. But a QDevice should be sufficient.

Great, thank you for the information! The config for Ceph shows:

osd_pool_default_min_size = 2   
osd_pool_default_size = 3

So that means at least 2 nodes to be up for Ceph to be working properly?

 

[UdoB]

So that means at least 2 nodes to be up for Ceph to be working properly?

Yes, two is acceptable for a short period of time. But it should be three!

Or more precise: two copies of a block must be available; being place on an OSD of two different hosts.

Disclaimer: I am not a Ceph specialist...