Optimal configuration of dual 10G connections

[Peter O'Kane]

Given some nodes with dual 10G Ethernet ports and a stack of 10G switches (4 switches stacked in full mesh at 40G) which is the better configuration?

1: Use one 10G port for ceph storage and the other for guest VM traffic so avoiding tagging overhead on the storage traffic. OR
2: Use both 10G ports trunked to a cross-node trunk on the switch stack with both ceph storage and guest VM traffic tagged on the trunk, gaining resilience against switch failure and potentially better storage bandwidth.

Nodes also have dual 1G ports allowing dedicated networks for management and corosync.

 

[mailinglists]

Personally I would use use option nbr.1. To have more redundancy.
Also, you do not have to use 802.3ad on Linux, there are many other bonding options, where you can more evenly balance the traffic, however you loose those nice LACP packets to test the links.

Please also note, that switches in a stack, still have a single point of failure, which is the software stack itself, I've experienced it first hand and upgrading the whole stack resulted in network reset due to a bug in software. So we use MC LAG ever since, and have completely independent switch nodes.