Proxmox\CEPH Hyper-Converged Networking

[Mike.S]

Hi,

I'm planning on reworking my cluster networking. Each of my nodes has (2) 1Gb NICs and (2) 10Gb NICs. I have (2) Dell 8124 Stackable switches.

Which would be the preferred HA method to configure the Proxmox Cluster/Corosync network? in a LAG or using the ring/link method?

I have read many times to use a separate physical network for CEPH and Proxmox Guest networks. Given that I have limited resources and 10Gb swithes, and that I want to obtain HA, I am contemplating bonding both 10Gb NICs, configuring CEPH network on it's own VLAN, and Proxmox Guest network on it's own VLAN. What are the issues with this configuration? It's my understanding that the LACP hashing algorithm will only allow a single TCP session to use only a single10Gb member port, not both at the same time. So bonding will not double CEPH network bandwidth to 20Gbs. The other10Gb port would receive the Proxmox Guest network traffic. Would it be correct to assume that this should prevent the Guest network and the CEPH network from interfering, unless of course there was a single port failure? What are the cons of this configuration?

 

[Alwin]

It is all about the interference of other traffic on shared media. Corosync needs guaranteed low and stable latency. Ceph performs better on low and stable latency as well. So you can imaging where its going with shared media.

 

[lucaferr]

I post our experience in case it can be useful:
We've run a 7 nodes Proxmox PVE v5 HA Cluster with Ceph for a couple of years. Each node has 5 NICs: 2 bonded (master-backup) 10G for LAN, 2 bonded (master-backup) 1G for WAN and last NIC is for running backups, on a separated network.
So Corosync and Ceph share the redundant 10G LAN network but, since Corosync is highly sensitive on latency and Ceph can push a lot of data on LAN interfaces in case of a rebalance due to a failed node (our storage is all NVMe, so network pressure can grow under particular circumstances), we prioritized Corosync using QoS rules, so in case of saturations it has precedence (DSCP tagging on UDP traffic going to ports 5404 and 5405 on each node and proper configuration on network switches). So far we're very happy about performance and how the cluster behaved when node failures occurred.

 

[Mike.S]

lucaffer, using DSCP tagging is a creative technique. Thanks for sharing. I'm going to use my two 1G NICs for corosync. I'm debating bonding versus the Proxmox ring/link method. I was hoping for opinions on which way to go.

I'm also seeking opinions on bonding the (2) 10Gb NICs and using the bonded interface for both the CEPH network and the Proxmox Guest network. My understanding is that the bonding hashing algorithm will distribute the Ceph network traffic onto one NIC and the Proxmox Guest network traffic onto the other NIC, essentially providing 10G for each, and providing HA in the event of a failure. Perhaps Alwin has some insight into this.

 

[spirit]

I'm also seeking opinions on bonding the (2) 10Gb NICs and using the bonded interface for both the CEPH network and the Proxmox Guest network. My understanding is that the bonding hashing algorithm will distribute the Ceph network traffic onto one NIC and the Proxmox Guest network traffic onto the other NIC, essentially providing 10G for each, and providing HA in the event of a failure. Perhaps Alwin has some insight into this.

if you use lacp with layer3+4 algorithm, traffic wil be balanced by srcip-srcport-destip-desport hash, so as ceph use 1connection by vm by osd, it'll balance really good on both links.

 

[Alwin]

I'm going to use my two 1G NICs for corosync. I'm debating bonding versus the Proxmox ring/link method. I was hoping for opinions on which way to go.

The ring/link expression is used by corosync. It is better to let corosync handle the failover than the underlying layers. Bonding provides failover on link-state changes, regardless if traffic can pass through.