Proxmox HA cluster with ceph - need help with network topology and storage

[osti]

Hello at all,

my plan is to build a 3 node ha cluster including ceph storage.
I have 3 server nodes, each one have 3 network cards 1. 2 x 25GbE / 2. 2 x 2GbE / 3. 2 x 10GbE.
I would like to seperate the cluster, storage and VM network like this:

Should it be ok, or is there a better solution (with my existing hardware)?
And the second question I have is regarding my storage. Each node have a bunch of harddisk attached to a raid controller (I know, that HBA would be better, but I can not change the hardware). The raid controller is in JBOD mode, I hope this is OK for ceph.
Each node have 6 SSD's ( 2 x 240GB, 4 x 1TB) and 12 HDD with 2TB each. Is there a ceph configuration guide how to use/group the disks?

Thank you so much, Osti

 

[czechsys]

ad 1]
a] 1 corosync link on ceph bond + 1 corosync link in mesh
b] 1 corosync link on 1gbps link on dedicated switch + 1 corosync link in mesh
c] 1 corosync link on 10gbps + 1 corosync link on 10gbps - all without mesh
etc...
tldr: split corosync link so they aren't dependent on one logical link

ad 2] yes, you can separate disks by device class eg. ssd/hdd. Check pve/ceph documentation.

 

[osti]

Hello czechsys,

thank you for the fast answer. But I am a little bit confused.
My goal was to seperate all 3 networks. You recommend to mix ceph and corosync(Cluster) network?
Or am I misunderstanding your answer?

 

[aaron]

My goal was to seperate all 3 networks. You recommend to mix ceph and corosync(Cluster) network?

I guess @czechsys was hinting at the fact that corosync itself can handle multiple networks (up to 8) so you don't need to have a redundancy layer underneath.

You can connect both NICs for corosync to (two) switches. Configure different subnets on each and also configure corosync to use both. If one becomes unusable, corosync will switch to the other by itself.

 

[osti]

You can connect both NICs for corosync to (two) switches. Configure different subnets on each and also configure corosync to use both. If one becomes unusable, corosync will switch to the other by itself.

OK, I can do this, but therefore I need another 2 switches.
My network drawing with the 2 mesh networks (for cluster and ceph) will not work?
Each node is connected to both other nodes. There should be no problem or?

What is the benefit of the solution with 2 more switches?

 

[aaron]

My network drawing with the 2 mesh networks (for cluster and ceph) will not work?
Each node is connected to both other nodes. There should be no problem or?

Should work as well. Depending on how you set up the full mesh network. For maximal redundancy, the RSTP variant is the one you want to use.

What is the benefit of the solution with 2 more switches?

A simpler stack for the Proxmox VE cluster network and therefore fewer chances of something going wrong

If you want to use HA for example, a stable corosync connection is of the utmost importance because it is used to determine if a node is still available.

How important is the network connection of the guests to the outside world? Ideally you also have two stacked switches for it in case a switch fails or if you just need to reboot one of the switches.

 

[osti]

ok, I have the possibility to use a second switch. Then I connect one port of each node-corosync to the first switch and also one port oft the node-vm-network to this switch.
And the second ports goes to the second switch. Then one switch will handle the cluster(corosync) network and the vm-network. The storage(ceph) network will be mesh like in the drawing.

 

[osti]

Hello, I updated my drawing. Would this be a better solution with 2 switches?

Thanks Osti.

 

[jsterr]

Do not bond Corosync, as it can increase latency, thats something you dont want because it can result in fencing (= rebooting) of nodes. Just use two single links, and you could use ceph link as a third fallback (if you want).