[SOLVED] Three-nodes Ceph mesh network with bond and VLANs

godzilla

Active Member
May 20, 2021
78
5
28
44
Hi everyone,

I'm designing a new 3-node cluster. It's not the first cluster I make but it's the first with a full-mesh Ceph network (no dedicated switch) . Each node has a double 10G NIC and they're already physically connected each other. So I've been reading the docs and I think the best solution for me could be Routed setup with fallback.

Anyway I was doing some reasoning: what if I create an active-backup bond and two VLANs on top of it? Something like this (let's say this is node 1):

Code:
### /etc/network/interfaces

# (cut)

auto bond1
iface bond1 inet manual
    bond-slaves eno49 eno50
    bond-miimon 100
    bond-mode active-backup
    bond-primary eno2
    bond-primary-reselect failure
    mtu 9000
#bond for Ceph HA

auto bond1.27
iface bond1.27 inet static
    address 172.27.0.1/24
#ceph-public

auto bond1.28
iface bond1.28 inet static
    address 172.28.0.1/24
#ceph-cluster

It could handle both connection and node loss easily, IMHO. Do you see any caveats? Too much overhead? Am I missing something? Would you still recommend broadcast bond over this?

Thank you!
 
Nevermind, I thought again about it and it's a bad idea™, because it would not handle a single connection loss. I would need two more network ports per node.