Adding a node to a cluster with Ceph storage

trilljester

Active Member
Oct 9, 2018
13
0
41
50
Hi there,

We have a cluster that has 4 nodes with Ceph storage. One of those nodes died yesterday, which puts the cluster in danger of freezing if another node goes down. I'm working on getting the dead node repaired and back online, but it will take a few weeks to get parts for it and get it fixed and running again.

In the meantime, could I add a server that does NOT have any storage on it just to keep the cluster going in case another node goes down? I have a couple of servers that are loaded in RAM and CPU but do not have any storage. If I could add one of those in there just to be able to run VM's, would that work?

And how would I go about adding that node to the cluster?
 
which puts the cluster in danger of freezing if another node goes down.
Are you talking about PVE Quorum? Just add a Quorum device: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support to get to an odd number of nodes to vote - as recommended for even numbers of nodes.

Are you talking about Ceph? Verify that it got re-balanced to store a copy of everything on each of the three survivors. (If you use the default size=3/min=size=2.)

And yes, of course you can add just another full blown server, with or without actual storage. But a QDevice should be sufficient.
 
Are you talking about PVE Quorum? Just add a Quorum device: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_corosync_external_vote_support to get to an odd number of nodes to vote - as recommended for even numbers of nodes.

Are you talking about Ceph? Verify that it got re-balanced to store a copy of everything on each of the three survivors. (If you use the default size=3/min=size=2.)

And yes, of course you can add just another full blown server, with or without actual storage. But a QDevice should be sufficient.

Great, thank you for the information! The config for Ceph shows:

Code:
osd_pool_default_min_size = 2   
osd_pool_default_size = 3

So that means at least 2 nodes to be up for Ceph to be working properly?
 
So that means at least 2 nodes to be up for Ceph to be working properly?
Yes, two is acceptable for a short period of time. But it should be three!

Or more precise: two copies of a block must be available; being place on an OSD of two different hosts.

Disclaimer: I am not a Ceph specialist...
 
There are 3 factors here:
  • Ceph quorum
Ceph quorum is maintaned by MON's: you need mayority of Ceph monitor running. Did the broken server had a mon? how many MON's do you have in that cluster? Post ceph status.​

  • Ceph I/O
If you have Ceph quorum, and if you are using the default config, 10 minutes after your server went down, their OSD got marked OUT and their copy of the data got recreated in the surviving nodes. Post ceph status to be sure about this. As long as all your PG are active you'll be able to read/write from Ceph. If you are unlucky to get another server down, you'll be left with just 2 copies of data but you will be able to read/write from them if you still have Ceph quorum.​

  • PVE quorum
You need majority of nodes up and running, so if you have only 2 from 4 total, you'll lose quorum. This means that you will not be able of doing almost no cluster actions besides shutting down VMs/LXCs (i.e. you will not be able to start a VM, create a new one, migrate to other node, etc). You could regain quorum for disaster recovery purposes issuing pvecm expected 2. Be careful and use such command if nodes are really down and you are fully sure that this is not just a network failure to avoid split brain situations.​
Note about QDevice: you can't add / remove a QDevice unless all cluster nodes are up and running. In your current situation this is not an option, but you may plan on adding one once your cluster is fully healthy.​
If you end up adding another node, make sure you don't reuse any hostname that may have ever existed in that cluster to avoid some SSH edge cases.
 
  • Like
Reactions: UdoB

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!