Proxmox Cluster unanswered questions

[ChESch]

Hello there,

We want to build a Proxmox-Cluster with 3 nodes, Ceph is used as hyperconverged storage.
We will use 3 Dell PowerEdge R710s for this, with an SFP+ Card installed (2 Ports), and an HBA-Adapter instead of the iPERC/6.

I still have some unanswered questions regarding this setup, tips are very appreciated!

So we will have an pfSense Firewall, then the 3 nodes connected to Network1, which is the "public" network with the IPs from the management-interfaces, VMs etc.

The SFP+ Cards will be configured in a Full-Mesh-Network for Ceph, so that every node is connected to the other 2.

I stumbled upon some posts which say that corosync should have it's own network as well... our problem is that we have no place for a switch anymore (public network will use switchports on the firewall directly), so my first question is, is it possible to use corosync3 in an Full Mesh configuration like ceph?

We will use 2 SSDs per Node in ZFS Raid1 as boot-drives, and for now add a single 4TB HDD in each node, which will be the OSDs. In the future, how should we configure ceph if we want to stock up the HDDs a bit, say 2 4TB HDDs per Node? Is it possible/wise to mix HDDs on a node, so let's say have 1 4TB and 1 2TB HDD per node, or even mix up storage size across nodes, e.g. node 1 total of 6TB, node 2 total of 8TB?

Third question, can we use ifupdown2 as drop-in-replacement so we do not have to restart every time the config is changed?

If you need any more information, feel free to ask, I hope my English is not to crude for you to understand.

All the best!

 

[troycarpenter]

Your cluster network doesn't need to communicate outside of itself. My first cluster used an off-the-shelf D-Link 1Gb consumer switch for the cluster traffic. I used a simple 192.168.200.0/24 network on one of the spare NIC ports on the nodes, and connected those to the 1Gb switch. No network connection to the rest of my network. Now I use 10Gb connections isolated by VLANs on my core switches for cluster traffic.

See:
https://pve.proxmox.com/wiki/Separate_Cluster_Network

 

[ChESch]

Your cluster network doesn't need to communicate outside of itself.

What do you mean with that?

Our problem is that we do not have place for another switch for the cluster-traffic (what i referred to as corosync), so I would like to make it Full Mesh like here: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server
I just don't know if that's a good idea when latency is of importance...