Proxmox HA cluster with ceph - need help with network topology and storage

osti

New Member
May 3, 2022
5
1
3
Hello at all,

my plan is to build a 3 node ha cluster including ceph storage.
I have 3 server nodes, each one have 3 network cards 1. 2 x 25GbE / 2. 2 x 2GbE / 3. 2 x 10GbE.
I would like to seperate the cluster, storage and VM network like this:
Proxmox_HA_Ceph_network.png
Should it be ok, or is there a better solution (with my existing hardware)?
And the second question I have is regarding my storage. Each node have a bunch of harddisk attached to a raid controller (I know, that HBA would be better, but I can not change the hardware). The raid controller is in JBOD mode, I hope this is OK for ceph.
Each node have 6 SSD's ( 2 x 240GB, 4 x 1TB) and 12 HDD with 2TB each. Is there a ceph configuration guide how to use/group the disks?

Thank you so much, Osti
 
Last edited:
  • Like
Reactions: itNGO

czechsys

Well-Known Member
Nov 18, 2015
328
28
48
ad 1]
a] 1 corosync link on ceph bond + 1 corosync link in mesh
b] 1 corosync link on 1gbps link on dedicated switch + 1 corosync link in mesh
c] 1 corosync link on 10gbps + 1 corosync link on 10gbps - all without mesh
etc...
tldr: split corosync link so they aren't dependent on one logical link

ad 2] yes, you can separate disks by device class eg. ssd/hdd. Check pve/ceph documentation.
 

osti

New Member
May 3, 2022
5
1
3
Hello czechsys,

thank you for the fast answer. But I am a little bit confused.
My goal was to seperate all 3 networks. You recommend to mix ceph and corosync(Cluster) network?
Or am I misunderstanding your answer?
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
2,919
461
88
My goal was to seperate all 3 networks. You recommend to mix ceph and corosync(Cluster) network?
I guess @czechsys was hinting at the fact that corosync itself can handle multiple networks (up to 8) so you don't need to have a redundancy layer underneath.

You can connect both NICs for corosync to (two) switches. Configure different subnets on each and also configure corosync to use both. If one becomes unusable, corosync will switch to the other by itself.
 

osti

New Member
May 3, 2022
5
1
3
You can connect both NICs for corosync to (two) switches. Configure different subnets on each and also configure corosync to use both. If one becomes unusable, corosync will switch to the other by itself.
OK, I can do this, but therefore I need another 2 switches.
My network drawing with the 2 mesh networks (for cluster and ceph) will not work?
Each node is connected to both other nodes. There should be no problem or?

What is the benefit of the solution with 2 more switches?
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
2,919
461
88
My network drawing with the 2 mesh networks (for cluster and ceph) will not work?
Each node is connected to both other nodes. There should be no problem or?
Should work as well. Depending on how you set up the full mesh network. For maximal redundancy, the RSTP variant is the one you want to use.

What is the benefit of the solution with 2 more switches?
A simpler stack for the Proxmox VE cluster network and therefore fewer chances of something going wrong ;)

If you want to use HA for example, a stable corosync connection is of the utmost importance because it is used to determine if a node is still available.

How important is the network connection of the guests to the outside world? Ideally you also have two stacked switches for it in case a switch fails or if you just need to reboot one of the switches.
 

osti

New Member
May 3, 2022
5
1
3
ok, I have the possibility to use a second switch. Then I connect one port of each node-corosync to the first switch and also one port oft the node-vm-network to this switch.
And the second ports goes to the second switch. Then one switch will handle the cluster(corosync) network and the vm-network. The storage(ceph) network will be mesh like in the drawing.
 

osti

New Member
May 3, 2022
5
1
3
Hello, I updated my drawing. Would this be a better solution with 2 switches?

Thanks Osti.
Info_Proxmox_VE_HA_v2.png
 

jsterr

Active Member
Jul 24, 2020
203
35
33
30
Do not bond Corosync, as it can increase latency, thats something you dont want because it can result in fencing (= rebooting) of nodes. Just use two single links, and you could use ceph link as a third fallback (if you want).
 
  • Like
Reactions: Christian St.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!