Cluster suggestion

[philliphs]

Hello Everyone.

I am upgrading the hardwares my cluster of 3 nodes. The specs are
3 identical nodes of:
Xeon i5 2487w
256GB RAM
500GB NVME x 1
500GB SSD x 2
1TB SSD x 1
2 port NIC10Gbe fiber running DAC cables (1 on VLAN for strictly ceph private network, 1 on separate VLAN for ceph public and proxmox public)
1Gbe NIC for corosync (connected to unmanaged standalone switch not connected to network)

I have few questions in regards to the setup:
1. With 3 nodes I can have redundancy of 1 node down. If I added 2 more devices for qdevice node, would it make my cluster redundancy into max 2 nodes down?
2. Since I'm planning to add the qdevices with just simple mini PC with 1 nic only, how would it work? Do I just connect to the corosync network, since my corosync network isn't connected to any running DHCP router etc.

Did anyone have similar experience?

 

[floh8]

With ceph its better not to get 2 Nodes down. Rather its better to have More than 3 Nodes in the cluster.

 

[philliphs]

With ceph its better not to get 2 Nodes down. Rather its better to have More than 3 Nodes in the cluster.

That's right, thus my plan to introduce extra 2 qdevice to make it into 5 pseudo nodes.

 

[gurubert]

You would need to have 5 nodes with Ceph OSDs to sustain 2 failed nodes. And even then they should not fail at the same time.

 

[philliphs]

You would need to have 5 nodes with Ceph OSDs to sustain 2 failed nodes. And even then they should not fail at the same time.

I see. Thank you for the answer. So adding 2 extra qdevices would not increase the cluster ability to sustain 2 failed nodes.

 

[LnxBil]

So adding 2 extra qdevices would not increase the cluster ability to sustain 2 failed nodes.

Not with CEPH, no. QDevice is for corosync and you need to satisfy the need for quorum for corosync and CEPH.

 

[UdoB]

You would need to have 5 nodes with Ceph OSDs to sustain 2 failed nodes.

https://docs.ceph.com/en/reef/rados/configuration/mon-config-ref/#monitor-quorum tells me there is a >50% consensus rule like in corosync Quorum, so we need also 5 MONs. Correct?

 

[philliphs]

https://docs.ceph.com/en/reef/rados/configuration/mon-config-ref/#monitor-quorum tells me there is a >50% consensus rule like in corosync Quorum, so we need also 5 MONs. Correct?

I think so. My guess is probably will need 2 pseudo nodes where these 2 will act just as corosync nodes and monitor. But I think these 2 nodes will also need 3 separate nics to be able to talk to the other 3 real nodes.

 

[LnxBil]

AFAIK, if only one node with working OSDs is left, the ceph pool will become read only, so you may also want to circumvent that.

 

[gurubert]

we need also 5 MONs

This is why I recommend a minimum of 5 nodes for Ceph.