(again a) network question regarding optimal setup

[OdinRagnarok]

I am building a proxmox cluster, which should go into production, with low cost hardware.
i have the following hardware:
3x dual xeon quadcore with 32gb ram, one ssd for proxmox, 4x ssd for ceph, 4x hdd for ceph, 2x 1gbit and 2x 10gbit network
2x dual 12core opteron with 32gb ram, one ssd for proxmox, 2x 1gbit and 2x 10gbit network (can be upgrade with 4x ssd for ceph)
1x 48 port 1ge and 2x 10ge switch
1x 48 port 10ge switch

So my idea is:
1ge: internet
1ge: corosync ring0
10ge: cluster network, corosync ring 1, ceph frontend
10ge: ceph replication

is that optimal?

 

[wolfgang]

Hi,

it is acceptable but not optimal for a perfect world ;-)
For an optimal setup, you need 2 more nics per node. Redundance on internet and second corosync ring.
If you do not use HA this will work perfectly.
When you use HA and the first ring0 of corosync will fail this can be a problem.
Because if a node failed Ceph will rebalance the data what will bring the network under load and this ends in increasing the latency what could a problem for corosync.

 

[OdinRagnarok]

Thx for the hints.
I thought having a pure 1ge link as primary corosync interface and as backup also on 10ge cluster interface if the 1ge switch fails, is ok.
Also i thought, that rebalancing will happen on the 10ge ceph replication network, not on the cluster interface.

 

[wolfgang]

Do you mean ceph replication=ceph cluster network?
http://docs.ceph.com/docs/master/rados/configuration/network-config-ref/
if so then you are right.
But then you have no redundancy in the ceph network.

 

[OdinRagnarok]

yes i only have one 10ge switch

So how would you install this cluster?
Installation with which interface? i planned to have internet via NAT on the 1ge corosync ring 0 network.