Move cluster operation from 1GB LAN to 10GB SFP+ ports

Fabio

Renowned Member
Jan 30, 2016
63
1
73
51
Good evening, let me start by saying that I am using Proxmox 8.1. I have a two node cluster, where I have enabled VM replication. The cluster communicates on 1GB LAN cards. The two servers also have SFP+ ports so I would like to use fiber cables. Moving the operation of the cluster and replicas to the connection on the SFP+ ports means changing the configuration of the sluster. I think this operation is delicate, do you think I could have any problems?
 
Instead of fiber, I would recommend a DAC cable, as you can't swap the pairs. But if you physically pay attention to this and simply change the interface in your vmbr, nothing should happen and if in doubt, just switch back to the old one.
 
I have had bad luck with SFP transceivers that provide for a copper interface. In my experience, they have a tendency to overheat and then fail intermittently. They work in principle, but in practice they are often more of a headache than what I am willing to deal with.

Fiber transceivers are a lot more forgiving from what I can tell, and you can cheaply buy pre-terminated fiber cables in a large variety of lengths. This is absolutely the way to go for connections within a single rack. A DAC cable addresses the same needs and could quite possibly be implemented over fiber. But you don't really know. And from my limited experience, DAC cables are a mixed bag. Some work fine, whereas others have intermittent problems similar to what I encountered with copper transceivers. I ended up ripping out all my DAC cables and things are much more stable now.

Things are a little less obvious for when you need to run cabling through the entire house. Fiber would be ideal here, but it is harder to deploy than copper. This is a discussion that you need to have with your electrician. But something like https://sschueller.github.io/posts/wiring-a-home-with-fiber/ is a good starting point.
 
I have had bad luck with SFP transceivers that provide for a copper interface. In my experience, they have a tendency to overheat and then fail intermittently. They work in principle, but in practice they are often more of a headache than what I am willing to deal with.
I can confirm that, I would definitely not recommend an RJ45 adapter.
And from my limited experience, DAC cables are a mixed bag. Some work fine, whereas others have intermittent problems similar to what I encountered with copper transceivers. I ended up ripping out all my DAC cables and things are much more stable now.
I only buy cables from Flexoptix, where I have a Flexbox and can brand the transceivers the way I need them. I've only had good experiences with the cables so far. Basically, if you buy cheap, you buy twice.
For the cabling in the rack I would always prefer DAC, everything that goes out should be normal SM or MM.
If it's also the Dell network cards, that shouldn't cause any problems. You have to be careful with other manufacturers or brands.
 
The cards are from. But at a configuration level, can I migrate from a cluster configured with 1GB cards to SFP+ cards? Can the operation cause problems?
 
I had the same idea. For redundancy reasons I tried to move the traffic on vmbr0 from 1x 1GbE (eno1) to a 2x 10GbE (bond0 alb), but afterwards the host could not reach its default gateway anymore, though other devices on the subnet could be pinged, except for the the other proxmoxes. In the end, I left the bond for the VMs and the 1GbE for the cluster.

I'm using RJ45 instead of fibre, but no SFP+ plugs.
 
Is it better to leave everything on 1GB? my cluster has only two nodes and the replicas are operated on 4 machines, one of which is windows. The disks are about 100GB each
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!