Hyperconverged Infrastructure with Proxmox and Ceph

[nitaish]

Hello All,

I am looking to build a hyperconverged infrastructure using Proxmox and Ceph Cluster. I need guidance on what should the infrastructure look like. Let me explain the use case.

I am planning to setup 3 Proxmox nodes with the following configurations

Dual Intel Xeon Octa Core Processors
256 GB DDR4 RAM
2 x 2 TB Enterprise SSD drives
4 x 4 TB Enterprise SATA drives
2 x 10 GbE Nics for storage network
2 x 1 GbE Nics for LAN

I want HA for each VM in the Proxmox nodes. I want to create two storage pools, one for the SSD drives and one for the SATA drives so that those who need performance would be hosted in SSD pool and those who look for storage, will be hosted in the SATA pool.

With the above configuration, I get the following raw resources available.

48 Physical/96 Virtual Cores
768 GB RAM
12 TB SSD storage
48 TB SATA storage

My questions are now

1. With Ceph Cluster and the HA setup, how much would be the usable resources for the production?
2. The guides available for Proxmox and Ceph setup, there is no method mentioned to create separate SSD and SATA pools. How can I achieve that?

- Nitesh

 

[Klaus Steinberger]

Ceph is very latency dependent. I would avoid a S-ATA only Pool, it is badly slow. Plan at least SSD journals for the S-ATA OSD's

It is possible to create separate pools, but you have to edit the Crushmap per Hand.

 

[nitaish]

I am planning to use the SSD drive on which the Proxmox will be installed as the journal. But the bigger question is how much usable resources will I get based on the above configuration.

 

[Klaus Steinberger]

With CEPH objects will be stored 3 times (Erasure Coding is currently not supported with proxmox as I remember)
Also you should not fill up pools with more than 60 percent, so you will have for images:

(12 TByte / 3 ) * 0.6 = 2.4 Tbyte SSD Storage
(48 TByte / 3 ) * 0.6 = 9.6 Tbyte SATA Storage

With newer operating systems (all actual linuxes, and Windows >= 8.1 >= 2012R2) you can use the Virtio SCSI Driver with discard option, so you will conserve some space. It depends of course on how full are the disks of the VM's will be, can be a factor of 2 to 2.5 from my experiences.

 

[nitaish]

What will be the usable storage if I go with 5 servers?

 

[Klaus Steinberger]

its the same "rule of three":

5 * 4 TB = 20 TB -> /3 = 6.7 TB * 0.6 = nearly 4 TByte

and same calculation for the SATA

 

[nitaish]

Considering the usable storage and the number of OSDs that will be created in each node (I assume 6 OSDs as there are in all 6 drives each node), what should be the ideal size of the Journal. Can I use a 240 GB Enterprise SSD or I need to go with at least a 960 GB Enterprise SSD?

 

[AlexLup]

I use 15gb SSD journals and for 6tb drives and I never go beyond 3gb for WAL+DB (Journal)...

 

[nitaish]

My question is still unanswered. Can I go with a 240 GB SSD for Journal per node or I need a larger one?