Home User to Corp IT Lab Environment


Active Member
Apr 27, 2019
Forgive me but I'm a network guy first and systems second.

I've been a home user for some time with a single node (no ceph, corosync, ha of any kind, etc) which is about as simple as it gets. I'm in the early phases of bringing up online nearly a 1/2 cabinet full of hardware that was used previously for openstack but it's been decommissioned. Just imagine someone plopping this gear in your lap and saying "here"....kinda where I'm at.

I haven't gotten into all the hardware details but there are 3 servers labeled "compute", 3 labeled "storage", and 3 labeled "control". Yea, I know this is greatly oversimplifying. For simplicity I'm just going to use the 3 compute & 3 storage chassis.

My high level plan is as follows
* Install PVE6 on everything
* Setup the datacenter
* Setup HA on the 3 compute hosts
* Deploy ceph

What I'm trying to understand is the proper networking as I think I have ample resources to do it right. My thoughts are as follow. There are vacant ports on each machine so I can modify this.
1x 1g for out of band to the servers...ipmi/idrac
2x 10g for ceph & corosync
1x 1g for the actual VM traffic to intranet/internet
So you got 9 servers and want to start using 3 of them?

Regarding the network, sounds okay except for one thing, having ceph & corosync on the same network. If you intend to go without redundancy and use 1x 10g for Ceph and the other 10g for Corosync then maybe use the second 10g for VM traffic?

Because corosync does not need a lot of bandwidth, what it needs is low latency. So in the case of having Ceph and Corosync on the same network you risk high latency in case Ceph saturates the network. In the second case you waste a lot of bandwidth.

Corosync can handle multiple links (`ringX` in the config). So you can configure fallback links.
It's actually more gear that now that I've got everything off the pallet. I'm just going to start with the compute (x4) & ceph (x3) nodes and will add the other nodes later. There is also 3 labeled control and 3 labeled build (2 have no drives). A couple nexus 9k switchs and a couple asr1k routers.

Thanks for the info on Corosync. I'll set that up so that it shares the 20g LAG (active active to two switches in MC-LAG) as well as the 1g for the VM external connectivity (goes to a different switch).

Is the proper order of operations as follow
  1. Install proxmox & updates
  2. Setup CEPH
  3. Setup cluster
  4. Setup HA


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!