Load balancing and redundancy

S

sj4318

New Member
Jul 18, 2024
5
0
1
Sydney
Hi all

I have a scenario and I'm stuck on which direction to take and would like some opinions on. I have done some digging around on the web and not found anything that satisfies the answer I'm looking for.

We have 3 Dell servers, each server has 4x10gb ports that I would like to use for the storage fabric. We also have 2 managed standalone 10gb switches to use in the fabric.

How would you configure this in a way that offers load balancing and redundancy?

Thanks :)
 
What do you want to use for Storage? Ceph? ZFS? Something else entirely?
 
shanreich said:
What do you want to use for Storage? Ceph? ZFS? Something else entirely?
Click to expand...
I intentionally left my question open ended but I should have specified the storage type! Apologies!!

We are wanting to implement Ceph. Each server has 12x4Tb HDDs, 2x1.92Gb NVMe for cache and a separate boot SSD. These servers were part of a VMware package... so now you know why I'm here lol.

Thanks :)
 
Just a note, if you use one NVMe for multiple OSDs then it’s a single point of failure for those OSDs.

I’m not clear on your question though…load balancing for your services, or for storage? Ceph handles the storage layer…
 
SteveITS said:
Just a note, if you use one NVMe for multiple OSDs then it’s a single point of failure for those OSDs.

I’m not clear on your question though…load balancing for your services, or for storage? Ceph handles the storage layer…
Click to expand...
I'm asking about the network config for the storage fabric. ie if you're going to run Ceph storage how would you configure the network based on 2 standalone managed switches and 4 10Gb ports out of each node. As far as Ceph and cache is concerned it will be 1 NVMe drive per 6 spindles and yep single point of failure much like it was with VMware so all good :)
 
well...
1] direct mesh network for ceph backend
2] if not LACP/MLAG/etc possible - STP or maybe any dynamic fast routing protocol
3] get LACP/MLAG switches
 
czechsys said:
well...
1] direct mesh network for ceph backend
2] if not LACP/MLAG/etc possible - STP or maybe any dynamic fast routing protocol
3] get LACP/MLAG switches
Click to expand...

I saw this earlier today after doing more digging posted by a retired Proxmox staff member. I think this is where I need to go assuming this will still work 6 years later:

K

Thread 'Bonding LAG with multiple switches'

Hi

I have a question about bonding and HA. I want to create a HA PVE cluster, but I am confused with the bonding and its mode. See this simplified picture. I have two switches (Mikrotik CRS317, not stackable) and multiple PVE nodes (just one signed).

What should I configure to create a HA network? The switches do support LACP, Active-Passive, etc. LAG protocols. But they don't support stacking nor multi-chassis LAG), so they are two separated L2 devices.

In PVE, I can choose the LACP mode, but am I right that I can not use this because I can not create a LAG group on the two different...

Scroll down, see the reply posted by Wolfgang re: nested bonding.

The switches I have are managed and stackable but to prevent them from rebooting simultaneously during firmware updates etc I have only uplinked them together and managed them individually, they are dedicated to the storage fabric. Previously they were dedicated to vSAN.

What do you think?

Thanks czechsys for your reply :)
 
VictorSTS said:
Unless you can stack or run MLAG to create LACP bonds between ports of those two switches and if you really want to add the bandwidth, go for full mesh routed with FRR using two LACP bonds per server as links among them [1].

If 10G per node is enough, simply use an active-passive bond on PVE and split two nics to both switches.

[1] https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_(with_Fallback)
Click to expand...
I'm thinking I'll try what was posted by Wolfgang. I can create LACP bonds on each switch then through nested bonding create an active/passive bond over the two LACP bonds.

Thanks Victor :)
 
VictorSTS said:
Unless you can stack or run MLAG to create LACP bonds between ports of those two switches and if you really want to add the bandwidth, go for full mesh routed with FRR using two LACP bonds per server as links among them [1].

If 10G per node is enough, simply use an active-passive bond on PVE and split two nics to both switches.

[1] https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Routed_Setup_(with_Fallback)
Click to expand...

If you're going with a routed setup via Openfabric / OSPF, then no bonds should be required - they're probably even detrimental to the whole setup. FRR supports ECMP, so just adding multiple interfaces to the same Openfabric router should already be sufficient for load balancing the traffic across multiple interfaces, as long as they have the same cost (which they do, unless explicitly set otherwise).
 
  • Like
Reactions: PwrBank and VictorSTS
You must log in or register to reply here.
Top Bottom