Redundant network with different speeds

CadilLACi

Active Member
Sep 20, 2016
15
0
41
42
Hi there!

I am looking for a way to redundantly configure CEPH for a new pve cluster.

The nodes have 1x10GBE and 2xGBE ports.

The nice way would be to have 2x10GBE ports per node and bond them together and make different VLANs for CEPH, cluster comm, separate cluster network, WAN, LANd, DMZ , etc....

Now how can I achieve something like this with 1 10GBE and 2 GBE ports? To make problems more difficult, i have one full 10GBE switch and one GBE switch. There is a GBe port on the 10GBE switch, so the 2 switches can send the same vlans through, but i can't do MC-LAG between them...

There are various budget constrains at work here...

I am supporting the comapny from 200 Km away so I would like to prepare for network problem shenigans......

What I came up with so far:

If I leave the 10GBIT ports unbonded and run every traffic through them via vlans, and then send the vlans through the gbit switch a trunked gbit port. On the Gigabit switch I have untagged ports for WAN, LAN and so forth.

Then in the gigabit switch I can LAG together the 2GBE ports of the servers. I can then send the separate cluster comm through these ports.

The only network redundancy I have this way is that the cluster communications wont fall apart if the 10GBE switch goes up in flames.

How can one do better?
 
I skimmed through you post, so this comment might not be most relevant, but MC LAG is only needed if you use 802.1ad. Linux kernel supports other bonding mechanism, that do not require (MC) LAG. See https://github.com/torvalds/linux/blob/master/Documentation/networking/bonding.txt and I also believe there are newer implementations available.

Try bonding mode 1 with 1 GB as backup:


Active-backup policy: Only one slave in the bond is
active. A different slave becomes active if, and only
if, the active slave fails. The bond's MAC address is
externally visible on only one port (network adapter)
to avoid confusing the switch.

In bonding version 2.6.2 or later, when a failover
occurs in active-backup mode, bonding will issue one
or more gratuitous ARPs on the newly active slave.
One gratuitous ARP is issued for the bonding master
interface and each VLAN interfaces configured above
it, provided that the interface has at least one IP
address configured. Gratuitous ARPs issued for VLAN
interfaces are tagged with the appropriate VLAN id.

This mode provides fault tolerance. The primary
option, documented below, affects the behavior of this
mode.
active-backup or 1
 
THX, I tried it and it worked!

I keep hitting a weird bug tho...

My GBE ports are eno1 and so on. MY 10GBE ports are enp65s0f0 like....

EVERY time I change something in the network, the active-backup writes the eno1 GBE port first. I have to manually change the order of the 2 slaves in the gui, and if I forget, the config sets the GBE port as primary. Even though I change something completely different in the network. It must be the GUI->config file translater that wasn't prepared for this, I guess.

PLS educate the script so it doesn't change the order of the slaves into ABC order.

Otherwise, it works like a charm!:D
 
Hi, I am glad I could help. Click a like to give me some credits.

Can you please copy paste your config and how it looks after PM changes it.
Then we can probably submit a bug request, referencing this forum post and it will get fixed.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!