New Hardware Configuration

[dthompson]

Hi all,

I'm hoping I can get some insightful input from this community. I'm looking at building out a new Proxmox cluster with Ceph as the backend. I'd like to get a couple of answers to see if this makes sense and if not where I should make changes.

http://bit.ly/2XvgrXa

Its a 2U 4 Node Super Micro Server.
I would like to boot and run PROXMOX off of an internal M.2 SSD and have the CEPH storage on each node with 6X2TB or 6X4TB SSD's

I have some questions:

Each node will have 2 10GB Ethernet ports. 1 for the public network and then 1 for the private Ceph and cluster configuration network. Will this work? Would this make more sense to have each server with 4 x 10GB Ethernet ports?

I ask this as I currently have the following but want to make sure this all works properly. I currently have a 1GB front end public facing network, 1GB private cluster configuration network, a 10GB network for storage mounts over NFS, and a 10GB network for my small CEPH (test) environment.

Can I have the networks as a 10GB network for public facing along with a combined 10GB for the cluster configuration and CEPH storage? Ideally I would have the cluster and Ceph networks VLANed into separate networks so there is no traffic interference. I would also like to my backup storage over NFS mounted 10GB VLAN as well with Jumbo Frames.

Does anyone see any issues with this type of deployment?

The VM's on the CEPH network will be primarily EMAIL (Axigen), PMG Servers, a couple of DNS Servers (BIND), Email Archive Server (Mailarchiva), a firewall (Kerio Control and or PFSense) and a document server.

This isn't high I/O based stuff going on here, but I'd like to make sure CEPH and this type of setup can handle the demands.

So with that, here is what I am thinking;

• 3 Nodes of the above configuration (It can do 4 but Ill keep it for expansion down the road)
• 256GB +/- RAM
• 2 X 10GB ethernet ports / server
• 1 X 128 and or 256GB M.2 SSD for Boot Drive
• 18 X 2TB and or 4 TB Samsung 850 SSD
• 2 X ??? INTEL CPU with X cores once I figure that out as well.

Can I get some input to say if this is the right way to go, or if I should rethink this. The thinking is that I'd like this to be as HA as possible along with keeping storage stable and expandable along with the performance by adding in more nodes as it grows.

Thank you for any constructive criticisms you can provide me here.

 

[sb-jw]

Its a 2U 4 Node Super Micro Server.

This is a SPOF at its best. If the Midplane have an issue, all Nodes will have it, if the Fans between the Disks / Nodes will fail, you could have an heavy overheating. If one PSU failed and failover to the other Power Circut, it is possible this fail too. If you put in some good Hardware, you have an heavy power consumption. You should only run this in a own Rack (not Shared) with 2 seperated power Circuts, where you know, the other Circut is not on his limit or in the near. Keep in mind, if one PSU failed all the load will go to the other one.

Each node will have 2 10GB Ethernet ports. 1 for the public network and then 1 for the private Ceph and cluster configuration network. Will this work? Would this make more sense to have each server with 4 x 10GB Ethernet ports?

It is enough to bond 2x 10GbE and use VLANs. Really, every Consultant will tell you, you need 2 for this, than 2 for this and in the end you will have around 16 Cables per Node - but this is bullshit. I maintain multiple Clusters with only 2x 10GbE and i can tell you, over years we dont see any issues regarding the NICs.

The VM's on the CEPH network will be primarily EMAIL (Axigen), PMG Servers, a couple of DNS Servers (BIND), Email Archive Server (Mailarchiva), a firewall (Kerio Control and or PFSense) and a document server.

If you only have the Server as you Cluster, i would recommend to setup an PfSense on another dedicated Server in HA Mode with an VM which running on the 2U Twin.

This isn't high I/O based stuff going on here, but I'd like to make sure CEPH and this type of setup can handle the demands.

If you use flash Only, i dont see any Problems here. But nobody can gurantee you this will not end in a disaster. You have to make sure what requirments you have on the Storage - but i dont think these services will make some trouble on this Setup. It might be enough to have 2 OSDs per Node and configure the Crush Rule to replicate per Node, not per OSD - but try it first, if it is not enough, you can plugin additional OSDs and grow up the Space and Performance.

• 1 X 128 and or 256GB M.2 SSD for Boot Drive

This is a SPOF, i would recommend to choose two Disks. As i can see in the Spec Sheet, it is supported.

2 SuperDOM support on the motherboard
1 NVMe or 2 SATA M.2 (22x80/60/42 mm)

• 18 X 2TB and or 4 TB Samsung 850 SSD

Really, do not use this for CEPH. You should use the PM883 or SM883, this are real Enterprise Grade SSDs, the 850er are Consumer.

 

[dthompson]

This is a SPOF at its best. If the Midplane have an issue, all Nodes will have it, if the Fans between the Disks / Nodes will fail, you could have an heavy overheating. If one PSU failed and failover to the other Power Circut, it is possible this fail too. If you put in some good Hardware, you have an heavy power consumption. You should only run this in a own Rack (not Shared) with 2 seperated power Circuts, where you know, the other Circut is not on his limit or in the near. Keep in mind, if one PSU failed all the load will go to the other one.


It is enough to bond 2x 10GbE and use VLANs. Really, every Consultant will tell you, you need 2 for this, than 2 for this and in the end you will have around 16 Cables per Node - but this is bullshit. I maintain multiple Clusters with only 2x 10GbE and i can tell you, over years we dont see any issues regarding the NICs.


If you only have the Server as you Cluster, i would recommend to setup an PfSense on another dedicated Server in HA Mode with an VM which running on the 2U Twin.


If you use flash Only, i dont see any Problems here. But nobody can gurantee you this will not end in a disaster. You have to make sure what requirments you have on the Storage - but i dont think these services will make some trouble on this Setup. It might be enough to have 2 OSDs per Node and configure the Crush Rule to replicate per Node, not per OSD - but try it first, if it is not enough, you can plugin additional OSDs and grow up the Space and Performance.


This is a SPOF, i would recommend to choose two Disks. As i can see in the Spec Sheet, it is supported.



Really, do not use this for CEPH. You should use the PM883 or SM883, this are real Enterprise Grade SSDs, the 850er are Consumer.

Hey thanks for the reply! You've given me some good stuff to think about and research more into. Perhaps I'll go in a different direction and look at dedicated individual or dual node servers that won't be a SPOF along with some external storage chassis that I can use for CEPH and grow as I need to.

I'm happy to hear that you are running 2X10GB in a LAG without any issues. I didn't even think that would be an option but if that works, then thats great, I'll look into that as well.

 

[wimp9849]

It is enough to bond 2x 10GbE and use VLANs. Really, every Consultant will tell you, you need 2 for this, than 2 for this and in the end you will have around 16 Cables per Node - but this is bullshit. I maintain multiple Clusters with only 2x 10GbE and i can tell you, over years we dont see any issues regarding the NICs.

Hi, regarding this: are you using the LAG used for ceph and vms traffic also for Cluster comunication (corosync) or is on 1 or 2 separate network?