three node Proxmox/Ceph solution

Oct 28, 2013
296
44
93
www.nadaka.de
Hi there,

it's not a real project (yet), but I would love to build a three node Proxmox/Ceph cluster in my mind and share the design ideas with you to hear your opinion about it.

Here are the requirements:
About 50 server VMs (full virtualization) - most of them windows servers, but also a bunch of linux VMs - will need about 8 TB of fast storage and about 300 GB of RAM. CPU requirements are moderate. Storage and RAM should be easy scalable. A redundancy of 2 would be fine, so 1 out of 3 machines is allowed to die.

I love simple, small and powerful concepts, so this is my hardware suggestion (three identical nodes):

Basis
Thomas Krenn 1HE Intel Dual-CPU RI2108+ Server
https://www.thomas-krenn.com/de/produkte/rack-server/1he-server/intel-dual-cpu/ri2108-plus.html

CPU
1x Intel Xeon® Processor E5-2690 v4 (14 Cores, 2,6 GHz)

RAM
256 GB (8x 32 GB ECC DDR4)

Storage
1x M.2 SSD 128 GB for the operating system and the software
1x 240 GB SSD (Samsung PM863a) for ceph monitor
3x 1,92 TB SSD (Samsung PM863a) as ceph OSDs

Networking
1x Intel X710-DA2 SFP+ Dual Port 10 GBit (1x 10 GBit for the VMs, 1x 10 GBit for Ceph)
(These NICs would be connected to some Juniper EX-3300 switches)


The questions a have at the moment, are:
1. Do you think that the storage backend will perform well? Especially referring to two performance critical Oracle databases and one Windows application server.
2. The mainboard has got some SATA connectors. Would it be fine to connect the SSDs to them, or is it recommended to use an HBA between the SSDs and the mainboard?
3. If anyone knows reliable non-US network cards, let me know. So far I only have experience with Intel stuff.

I'm so curious about your opinion!

Thanks a lot and many greets
Stephan
 
Hi,

with the SM863a we get about 400MB in a single VM.

see

I'm not sure about the OracleDB.

SATA on board controller are fine.
 
Hi Wolfgang,

thanks for your reply!

Hi,
with the SM863a we get about 400MB in a single VM.
that sounds good! And there is another useful hint in your answer: The SM-series seems to fit better to database-like workload than the PM-series.

I'm not sure about the OracleDB.
Do you have benchmark results with some 4k random tests? I would like to see the IOPS of such a workload.

SATA on board controller are fine.
Fine, thanks!

Another question: How much RAM should be reserved for ceph in such an environment?

Thanks and greets
Stephan
 
Do you have benchmark results with some 4k random tests? I would like to see the IOPS of such a workload.

I have no 4k io benchmarks and the cluster is at the moment not setup for tests.

Another question: How much RAM should be reserved for ceph in such an environment?

calculation are is
1 GB per osd
4 GB per mon
1 GB per used TB disk on each node
 
I would ask on the ceph list. I think they can give you more information about DB on Ceph.
 
Hi there,

a question regarding the support subscriptions: Let's say, we will build up this cluster using the "Proxmox" way: Default Proxmox VE installation from ISO and using pveceph for Ceph installation. And let's assume that we buy standard support subscriptions for the environment. And one day we run into issues concerning Ceph. Will the Proxmox team try to help us? Or will they say: "Sorry, not our department."

Oh, and another one: If we like to do some performance tuning for Ceph: Would Proxmox team support us? Could we use a support ticket for this?

Thanks and greets
Stephan
 
Hi there,

a question regarding the support subscriptions: Let's say, we will build up this cluster using the "Proxmox" way: Default Proxmox VE installation from ISO and using pveceph for Ceph installation. And let's assume that we buy standard support subscriptions for the environment. And one day we run into issues concerning Ceph. Will the Proxmox team try to help us? Or will they say: "Sorry, not our department."

Oh, and another one: If we like to do some performance tuning for Ceph: Would Proxmox team support us? Could we use a support ticket for this?

Thanks and greets
Stephan

If you have questions regarding the details of the support subscriptions, please contact office@proxmox.com
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!