Planning PVE/CEPH cluster - recommendation

[ZS-Man]

Hi all,

I am planning new HA Proxmox/Ceph cluster ( on Supermicro HW)
Can anybody see this setup and tell me if there will be any performance problem?

2x CEPH only nodes:
1xCPU Xeon 2603v3 1.6 GHz
32GB RAM
2x10GB for storage network
2x1GB for LAN
This nodes will be used only for CEPH

1x combined CEPH/Proxmox VM node
2x CPU Xeon 2620v3 2.4GHZ
64GB RAM
2x10GB for storage network
2x1GB fot LAN

HDD: In all nodes I am planning 3 (or 4) x 600GB 10k SAS and 1xIntel 3610 SSD for journal

On combined node will run only any little VM (2-4GB RAM, 1-2 vCPU)

I can add another node, no CEPH, no HDD(OSD), 2x 10GBit for storage network, only for running big VMs (2 - 3 VMs - jBOSS, Pentaho, Zimbra...)

For better performance I can add 2 x 400GB SSD to all CEPH nodes as another storage pool and use it for this "big" VMs

All OSD will be connected to HBA, journal SSD to onboard SATA.

My question:
1. For HDD pool - is using 10k SAS ok? or 7k wil be sufficient ? Will be there noticeable performance difference?
2. Both HDD and SSD pool on one CEPH cluster? is this possible?
3. HBA controller - AOC-S3008L-L8e(3008) SAS3HBA (ITmode) or AOC-S3008L-L8I(3008) SAS3HBA (IRmode) ? What will be better?

Thanks.
(and sorry for my English... )

 

[manu]

Hi Zs Man
If you're not sure about which server specs would have the best specs for PVE, I would advise you to have a look at the offers from those guys:
http://www.osnet.eu/fr/content/proxmox-superserver and maybe add a proxmox basic support license on top of that

getting best performances without sacrifying data safety is a never ending topic

 

[udo]

Hi all,

I am planning new HA Proxmox/Ceph cluster ( on Supermicro HW)
Can anybody see this setup and tell me if there will be any performance problem?

2x CEPH only nodes:
1xCPU Xeon 2603v3 1.6 GHz

Hi,
I would use an faster CPU, like the E5-1650v2/3 with 3.5 GHz

32GB RAM
2x10GB for storage network

more ram - more speed (caching, but depends on your workload).
use different networks for ceph-storage (public_network) and ceph-replication (cluster_network).

2x1GB for LAN
This nodes will be used only for CEPH

1x combined CEPH/Proxmox VM node
2x CPU Xeon 2620v3 2.4GHZ
64GB RAM
2x10GB for storage network
2x1GB fot LAN

HDD: In all nodes I am planning 3 (or 4) x 600GB 10k SAS and 1xIntel 3610 SSD for journal

On combined node will run only any little VM (2-4GB RAM, 1-2 vCPU)

I can add another node, no CEPH, no HDD(OSD), 2x 10GBit for storage network, only for running big VMs (2 - 3 VMs - jBOSS, Pentaho, Zimbra...)

For better performance I can add 2 x 400GB SSD to all CEPH nodes as another storage pool and use it for this "big" VMs

All OSD will be connected to HBA, journal SSD to onboard SATA.

normaly the onboard sata are slower than hba-connected (depends on hba/driver/...).

My question:
1. For HDD pool - is using 10k SAS ok? or 7k wil be sufficient ? Will be there noticeable performance difference?

I don't have experiences with SAS-OSDs, but I assume you notice the difference between 10k-SAS to 7k-SATA.

2. Both HDD and SSD pool on one CEPH cluster? is this possible?

yes, you can define different pools - but need to manualy change the crushmap.

3. HBA controller - AOC-S3008L-L8e(3008) SAS3HBA (ITmode) or AOC-S3008L-L8I(3008) SAS3HBA (IRmode) ? What will be better?

Don't use the raid-mode!! Esp. with LSI you need than such things like raid-1 with 1 drive... crapware!! This was my first change in the ceph-cluster to remove the LSI-Controller which run in IR-mode only.

Test your config to see, that the speed meet your vision!

Udo

 

[ZS-Man]

Hi, thanks you very much for your reply.

I cannot make any test, I am in planning phase before buying HW...... and budget is limited..

CPU - big deal with sockets vs. cores vs. HT vs. HZ ... So from what CPU config will CEPH node profit better? eg. 2x 260Xv3 1.6 (or1.9)GHz vs. 1x 1650v2/3 3.5 GHz ?

HDD - I plan 10bay case:
for OS: 2x SuperDOM 16GB in RAID1 onboard
for OSD (standart pool): 4x 600GB 10kSAS2 - connected to HBA .
for OSD (fast pool): 2x400GB SSD SATA - connected to HBA
for journal : 1x 200 GB SSD SATA - connected to onboard SATA
3 bays will be empty for adding disk and expand pool in future.

HBA - OK, I will use AOC-S3008L-L8e(3008) SAS3HBA (ITmode) (or another ITmode HBA)


Anybody knows about benchmark / performance comparsion with 10k SAS vs. 7k SATA in small CEPH cluster?

Thanks
ZS-Man