Nodes interconnection via fiber optics

mmaridev

Member
Jun 4, 2020
10
0
6
Hi all,

I have to admit, this is a weird question. I need to setup a new Proxmox cluster that, using ZFS, will be replicating VMs and CTs disks through different nodes.
I know that quite a lot of bandwith might be required so I was checking out Mellanox with a 10Gbps SFP+. The problem is that purchasing a switch with 20 SFP+ ports becomes really expensive. Therefore, I was tinkering of cheaper solutions.
I realized i could simply interconnect the servers, since every one has two SFP+ ports, and then use one as uplink. Very straightforward:

switch (SFP+) -> (SFP+ port 1) node 1 (SFP+ port 2) -> (SFP+ port 1) node 2 (SFP+ port 2) -> (SFP+ port 1) node 3 etc...

Correct me if I'm wrong but bridging with a virtual bridge the SFP+ ports on the servers should do the trick here (Q: does the traffic going e.g. from the switch to node 2 need to pass from the CPU of node 1?).
Anyway, this idea has the problem that I'm deploying this solution to enhance fault rendurance and here, if one of the nodes, or worse, the first node with the uplink, fails, all other nodes in cascade would go offline. I need an idea to get this solution fault rendurant. I was thinking of "closing the chain" so connect the last node back to the switch (or better, to another switch... also switches can fail) but I can't figure out how should this be configured via software: If i set it as a virtual bridge, traffic will collide while going to the switch from both the first and the last node whereas configuring it as a bond would not allow other nodes to benefit of the connection he's uplinking from the switch in case the other one fails.
Is this even technically possible? And is it possible in Proxmox? Should I configure the switch(es) in a specific manner? (e.g. link aggregation?)

Thanks in advance,
Best,
Marco
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!