CEPH client only node

[kissze]

Hello,

I have a hyperconverged Proxmox+CEPH cluster witch is working well. I have 2 networks currently (i know it would be better to separate ceph public from vm traffic, but currently i only have 4x10g per node), 1 for ceph cluster on a dedicated bond and 1 for ceph public and VM traffic on a dedicated bond. Now i have a fourth server and i want to join to Proxmox cluster, but not to the CEPH cluster, because it no makes sense with the 2/3 replicas configured. Can i join the new node without adding the ceph cluster network to it to work as a client only with another three nodes?
Or i have to configure the cluster network also to work everything as on the other nodes?

Cheers,
Zoltan

 

[alexskysilk]

Can i join the new node without adding the ceph cluster network to it to work as a client only with another three nodes?

Joining a PVE cluster does not mean you're joining the ceph cluster. IF you had the right connectivity it would just work, but its fine if you dont.

Just make sure to NOT include that node in workloads that involve the ceph store.

 

[kissze]

Joining a PVE cluster does not mean you're joining the ceph cluster. IF you had the right connectivity it would just work, but its fine if you dont.

Just make sure to NOT include that node in workloads that involve the ceph store.

Yes, i know that these things are different. But with default install there is no CEPH installed i think. So i can join the PVE cluster, but what will happen with ceph storage? I have to install manually the packages for CEPH or install from the GUI? As far as i remember if im installing from the gui, then some configuration will be created by PVE and the question is that how can i tell the GUI to use only the public network, no cluster network? Maybe its confusing, but i hope its understandable

 

[alexskysilk]

But with default install there is no CEPH installed i think.

from the ceph repo. ceph client packages are provided with debian.

So i can join the PVE cluster, but what will happen with ceph storage?

Nothing. Why would you expect it to?

how can i tell the GUI to use only the public network,

just enable and install the proxmox ceph repo (pveceph install) so it can use the ceph configuration provided in /etc/pve (easiest way.) it would just work.

 

[kissze]

from the ceph repo. ceph client packages are provided with debian.


Nothing. Why would you expect it to?


just enable and install the proxmox ceph repo (pveceph install) so it can use the ceph configuration provided in /etc/pve (easiest way.) it would just work.

Thanks, i will try this method (pveceph install)

I mean what will happen with the storage defined in the pve cluster. Will show up as unreachable for example.

I know that there will nothing happens with the ceph storage/pool itself

 

[UdoB]

i want to join to Proxmox cluster, but not to the CEPH cluster, because it no makes sense with the 2/3 replicas configured.

Why doesn't it make sense???

Four nodes are better than three ;-)

Beside the capacity aspect think: with three Ceph-Nodes and one failing there is NO CHANCE for self-healing as the "failure domain" is "node".

But with four Ceph-Nodes, still with size=3/min_size=2, self-healing will be possible! When one node fails the required three are still available.

(( Of course only one node may fail. This was true for three nodes and still is true for four nodes. ))

Some more aspects in this regard: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/


Also note that PVE-quorum and Ceph MON majority is completely independent. For PVE you may add a separate "QDev" to get five votes, for Ceph such a thing does not exist.

 

[kissze]

Why doesn't it make sense???

Four nodes are better than three ;-)

Beside the capacity aspect think: with three Ceph-Nodes and one failing there is NO CHANCE for self-healing as the "failure domain" is "node".

But with four Ceph-Nodes, still with size=3/min_size=2, self-healing will be possible! When one node fails the required three are still available.

(( Of course only one node may fail. This was true for three nodes and still is true for four nodes. ))

Some more aspects in this regard: https://forum.proxmox.com/threads/fabu-can-i-use-ceph-in-a-_very_-small-cluster.159671/


Also note that PVE-quorum and Ceph MON majority is completely independent. For PVE you may add a separate "QDev" to get five votes, for Ceph such a thing does not exist.

Actually i have calculated originally with three nodes. In case of node failure, we dont have to “suffer” because of instant rebalancing, and we have spare machine.

After your comment i started to think about to add the fourth node to the ceph cluster, but currently i dont know that what will happen if a node goes down.
In the other hand the 3 replica on 4 node is more complicated regarding “where is my data?” question

Currently i have 3 osds per node (enterprise ssds), can i add the fourth node with all of their disks at once?
For example, i can buy another 3 enterprise ssd, join the ceph cluster, add a mon, then can i add all of those ssds at once? As i remember i had to add one by one, and the rebalancing started with every new osd. I would like to avoid this if were going to this way instead of putting more osds to the existing nodes.

 

[VictorSTS]

Actually i have calculated originally with three nodes. In case of node failure, we dont have to “suffer” because of instant rebalancing,

With just 3 nodes, if one fails no rebalancing will happen. There will be some rebalancing if one OSD fails, though. Check this thread with some detailed expanations about this [1].

and we have spare machine

Add it to the cluster then with OSD's and let Ceph deal with failures

currently i dont know that what will happen if a node goes down

After 10 minutes, all the "third replicas" stored in that node OSD's will be recreated in the remaining OSD's of the cluster.

“where is my data?” question

In the Ceph cluster, with 3 replicas stored in 3 OSD each one in a different host. The asnwer to that question is somewhat irrelevant.

can i add the fourth node with all of their disks at once?

Yes. Enable norebalance, norecovery, nobackfill OSD flags (from WebUI or CLI), create as many OSD you want, then disable the flags and the rebalancing will start to all 3 newly added OSD.

join the ceph cluster, add a mon,

No: Ceph requires an uneven number of mon (remember, it uses Paxos too as PVE cluster, but does it's own quorum). In PVE simply use 3 or 5 mons. You would simply add OSDs in the new node. In fact, you could also simply install Ceph packages, not deploy any OSD and use it as a Ceph "client" and access the Ceph storage created by the other 3 hosts Ceph cluster.

I would like to avoid this if were going to this way instead of putting more osds to the existing nodes.

Backfill and rebalance is done with very very low priority by default, it will have negligible performance impact in your cluster. I would not worry about that.


[1] https://forum.proxmox.com/threads/u...-node-cluster-with-12-osds.161734/post-745316

 

[kissze]

Were moved to the 4 node setup, thank you guys. Also i have tested the "client" only join and yes, its working without any configuration, i only had to join to the PVE cluster and the CEPH storage which is defined in the cluster get available on the new node instantly.