Home User to Corp IT Lab Environment

[MrPaul]

Forgive me but I'm a network guy first and systems second.

I've been a home user for some time with a single node (no ceph, corosync, ha of any kind, etc) which is about as simple as it gets. I'm in the early phases of bringing up online nearly a 1/2 cabinet full of hardware that was used previously for openstack but it's been decommissioned. Just imagine someone plopping this gear in your lap and saying "here"....kinda where I'm at.

I haven't gotten into all the hardware details but there are 3 servers labeled "compute", 3 labeled "storage", and 3 labeled "control". Yea, I know this is greatly oversimplifying. For simplicity I'm just going to use the 3 compute & 3 storage chassis.

My high level plan is as follows
* Install PVE6 on everything
* Setup the datacenter
* Setup HA on the 3 compute hosts
* Deploy ceph

What I'm trying to understand is the proper networking as I think I have ample resources to do it right. My thoughts are as follow. There are vacant ports on each machine so I can modify this.
1x 1g for out of band to the servers...ipmi/idrac
2x 10g for ceph & corosync
1x 1g for the actual VM traffic to intranet/internet

 

[aaron]

So you got 9 servers and want to start using 3 of them?

Regarding the network, sounds okay except for one thing, having ceph & corosync on the same network. If you intend to go without redundancy and use 1x 10g for Ceph and the other 10g for Corosync then maybe use the second 10g for VM traffic?

Because corosync does not need a lot of bandwidth, what it needs is low latency. So in the case of having Ceph and Corosync on the same network you risk high latency in case Ceph saturates the network. In the second case you waste a lot of bandwidth.

Corosync can handle multiple links (`ringX` in the config). So you can configure fallback links.

 

[MrPaul]

It's actually more gear that now that I've got everything off the pallet. I'm just going to start with the compute (x4) & ceph (x3) nodes and will add the other nodes later. There is also 3 labeled control and 3 labeled build (2 have no drives). A couple nexus 9k switchs and a couple asr1k routers.

Thanks for the info on Corosync. I'll set that up so that it shares the 20g LAG (active active to two switches in MC-LAG) as well as the 1g for the VM external connectivity (goes to a different switch).

Is the proper order of operations as follow

1. Install proxmox & updates
2. Setup CEPH
3. Setup cluster
4. Setup HA