First Proxmox HA cluster with 3 nodes and 2 ceph pools

Bartek

New Member
Oct 22, 2023
1
0
1
Hi,
In the company we have decided to migrate our applications from a public cloud provider to our own solution based on proxmox ha cluster. Since this is the first such installation I have a great request for an opinion on the approach we have chosen for this.

General assumptions:
- 3 identical servers (cpu, ram, storage) put together in ha cluster
- servers hidden behind 2 fortigate 60f firewalls

Hardware we plan to use i 3 identical serwers:
DELL R730xd with 2x rj45 1G, 2x rj45 10G
2x Intel Xeon processor E5-2690 V4 SR2N2 2.6-3.5 GHz
4x 64GB RAM DDR4
2x 400GB SSD
2x 12TB SAS
1x 4TB SSD
and 2 Fortigate 60F

The current plan is to:
1. connect the servers in a mesh cluster using 10G links, and 1G links with fortigate set to active-pasive mode
2. put the system on 400GB SSD drives tied together in RAID1
3. set up 2 ceph pools. The first pool (2x12TB HDD SAS) is to be dedicated for functions that require a lot of space, but are rarely used (low IOPS). The second (4TB SSD) dedicated for VMs operating system and applications with a high number of IOPS

My question to you is:
1. Will this solution generally work well as a production environment. Our VMs will host several web platforms, and databases. The whole thing is rather small, the growth in terms of data is about 4TB per year. Data transfer to and from platforms is probably around 500GB per month?
3. Is mesh cluster a good idea? Certainly beneficial due to the lack of need to buy expensive switches.
4. Will 10G connections on rj45 for ceph needs be sufficient, or is it better to think of faster connections now?
5. Whether such a solution can be reasonably scaled in the future. In terms of RAM or disks I know that there should be no problems, but what if you need to supply another server (or servers to keep an odd number of them). Would a mesh cluster still be a good solution in such a case?

Thanks for any feedback
Bartek
 
Hello,

I am only going to comment on a few items

3. Yes mesh networks can be a good idea, as long as you know you won't be adding more nodes in the foreseeable future.
4. It heavily depends on your usage, I would suggest to get 25G NICs for Ceph traffic if possible
5. If you plan to add more nodes in the future you will definitively need switches. Do note that for a small 3-node cluster I would recommend to use either one OSD per device class, or at least four.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!