Network suggestion for 3 node cluster with 4 NICs

Jun 26, 2019
Hi there,

I want to build a 3 node cluster with ceph installed locally on each node. Unfortunately my Hardware is limited to 2x1G and 2x10G Interfaces. In my test environment I used the 10G interface in active/passive bond configuration, 1x1G for cluster network and vm2vm communication and 1x1G for uplink.
The Problem: no redundancy for cluster network and vm2vm communication. (uplink is solved with one virtual gateway routers on each node running vrrp). If the cluster network interface fails, the node looses quorum and VMs running on that node can't reach the others.
There is redundancy in the 2x10G interfaces, but everyone says "don't run corosync with storage traffic". Bonding the 2x1G and sharing uplink with cluster traffic is also not a good idea, I think.
An other option I was thinking of:
Using the 10G bond for storage and vm2vm, keep corosync on a single 1G and configure a 2nd ring for corosync on the 10G bond. This will keep a dedicated 1G link for corosync, but share the redundant link only if the other one fails.
Any other suggestions?

Best regards,
Any other suggestions?

What about the obvious: Add another dual 10 GBE adapter to each node and run it like it should be.

I don't see that many problems with corosync and storage on 10 GBE with respect to running one of them "only" in a non-redundant setup. Yes, it's not optimal, but it'll work and you will not have a SPOF.
Hi LnxBil,

thanks for your reply. Unfortunately adding another dual 10 GBE adapter is not possible due to Hardware and Provider restrictions but we managed to get another pair of 1G interfaces into the Systems which should be fine as well.

Best regards,


The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!