Optimal configuration of dual 10G connections

Peter O'Kane

Active Member
Feb 21, 2019
3
0
41
66
Given some nodes with dual 10G Ethernet ports and a stack of 10G switches (4 switches stacked in full mesh at 40G) which is the better configuration?

1: Use one 10G port for ceph storage and the other for guest VM traffic so avoiding tagging overhead on the storage traffic. OR
2: Use both 10G ports trunked to a cross-node trunk on the switch stack with both ceph storage and guest VM traffic tagged on the trunk, gaining resilience against switch failure and potentially better storage bandwidth.

Nodes also have dual 1G ports allowing dedicated networks for management and corosync.
 
Personally I would use use option nbr.1. To have more redundancy.
Also, you do not have to use 802.3ad on Linux, there are many other bonding options, where you can more evenly balance the traffic, however you loose those nice LACP packets to test the links.

Please also note, that switches in a stack, still have a single point of failure, which is the software stack itself, I've experienced it first hand and upgrading the whole stack resulted in network reset due to a bug in software. So we use MC LAG ever since, and have completely independent switch nodes.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!