Looking for a "cheap" way to upgrade to 40gibt
- Thread starter proxwolfe
- Start date
Dual ConnectX-3 cards are stupid cheap and support 40gb ethernet or infiniband depending on what they are currently flash to. The quad port cards are pretty much x4 the price as the dual port cards, so really depends on the price per port you are looking to spend. The expensive part of the whole thing is going to be switch. If you got a small handful of servers, you can skip the switch and go direct attached from one to another.
I do not know of any NIC cards that support RJ45 over 10gb. You can use QSFP+ cards with Transceiver and Cat8e cables to do 40gb RJ45, but I am pretty sure that will be more expense than just using QSFP+ cables.
I do not know of any NIC cards that support RJ45 over 10gb. You can use QSFP+ cards with Transceiver and Cat8e cables to do 40gb RJ45, but I am pretty sure that will be more expense than just using QSFP+ cables.
Thank you for this suggestion. These cards are indeed ridiculously cheap. I shall investigate this further. My first impression is that Infiniband is the better technology - would you agree?
I do, in fact only have a small home lab cluster of three nodes. But I am a bit reluctant to wire them directly without a switch because I fear that I would mess up the configuration. But I might change my mind when I see the price tag of a switch...
I do, in fact only have a small home lab cluster of three nodes. But I am a bit reluctant to wire them directly without a switch because I fear that I would mess up the configuration. But I might change my mind when I see the price tag of a switch...
To chime in, i do recommend running infiniband, you can use IPoIB which will basically act as a normal network card.
the cx3 cards that were recommended do work pretty good and are usually around $20 per for a dual port.
you can get a sx6036 infiniband switch for dirt cheap, around $100-$150 and i would highly recommend one for a home lab.
there are some slight issues being that you will need to update the firmware to the latest which can take some effort, and you will need mellanox branded IB cables, but other than that its a fairly plug and play solution.
RDMA alone is a good reason to get IB.
the cx3 cards that were recommended do work pretty good and are usually around $20 per for a dual port.
you can get a sx6036 infiniband switch for dirt cheap, around $100-$150 and i would highly recommend one for a home lab.
there are some slight issues being that you will need to update the firmware to the latest which can take some effort, and you will need mellanox branded IB cables, but other than that its a fairly plug and play solution.
RDMA alone is a good reason to get IB.
Is bandwidth fever a thing? Because I think I'm feeling it 
But I also found sx6015 and sx6025 that are quite affordable. Main difference, apart from the number of ports on the sx6015, seem to be managed vs unmanaged. Would I need the managed version when I want to partition the switch into separate networks? I currently have three networks (Proxmox cluster, Ceph cluster and PBS) and I wouldn't want to set up three switches.
When googling for the differences, I happened upon a discussion about used switches and someone pointed out that they may come without ethernet license. Is that something I would need to look out for, if I want to use (only) IPoIB (or is that in relation to using the switch as an ethernet switch itself)?
A concern is that these switches come with 2 300W PSUs. Would you be able to give me a rough estimate of the power consumption?
And lastly, some of this equipment is described as "40gbe", some is "40gbe/56gbe" and some is "56gbe" - is that all the same (possibly depending on the quality/standard of the cables) or are these different (incompatible) standards and I need to make sure to get all parts with the same designation?
Thanks!
With a quick check on ebay, I found a couple sx3036 but, alas, at substantially higher prices. Probably need to watch that space and strike in a good moment...
But I also found sx6015 and sx6025 that are quite affordable. Main difference, apart from the number of ports on the sx6015, seem to be managed vs unmanaged. Would I need the managed version when I want to partition the switch into separate networks? I currently have three networks (Proxmox cluster, Ceph cluster and PBS) and I wouldn't want to set up three switches.
When googling for the differences, I happened upon a discussion about used switches and someone pointed out that they may come without ethernet license. Is that something I would need to look out for, if I want to use (only) IPoIB (or is that in relation to using the switch as an ethernet switch itself)?
A concern is that these switches come with 2 300W PSUs. Would you be able to give me a rough estimate of the power consumption?
And lastly, some of this equipment is described as "40gbe", some is "40gbe/56gbe" and some is "56gbe" - is that all the same (possibly depending on the quality/standard of the cables) or are these different (incompatible) standards and I need to make sure to get all parts with the same designation?
Thanks!
heres an amazon listing for the sx6036, i bought 2 from this place and they worked great:
Mellanox sx6036
heres one for ebay thats cheaper:
Mellanox sx6036
you would for sure want a managed one, infiniband needs a subnet manager and they come installed on managed switches. if you dont have a managed switch then you would need to install and configure opensm on your hosts and its not worth the difference in price. you can set up separate partitions in the managed switch.
the IB switches do not come with ethernet capability by default and only do infiniband out of box. you can get an ethernet license but i havnt found it to be all that helpful and i just instead use IPoIB which allows me to use IB how i want it as well as standard ethernet traffic without having to buy a license.
the switch consumes about 50w with 4 nodes on it.
the switch supports fdr10 (40gb) and fdr(56gb). to use fdr you would need a cx3 pro, or a cx4 fdr/edr card. you will also need a mellanox branded cable to support that. this particular switch doesnt unlock that till the firmware is updated to the lastest available, it previously was locked behind a license, but they have since allowed it to operate at 56 for free with a newer firmware.
Mellanox sx6036
heres one for ebay thats cheaper:
Mellanox sx6036
you would for sure want a managed one, infiniband needs a subnet manager and they come installed on managed switches. if you dont have a managed switch then you would need to install and configure opensm on your hosts and its not worth the difference in price. you can set up separate partitions in the managed switch.
the IB switches do not come with ethernet capability by default and only do infiniband out of box. you can get an ethernet license but i havnt found it to be all that helpful and i just instead use IPoIB which allows me to use IB how i want it as well as standard ethernet traffic without having to buy a license.
the switch consumes about 50w with 4 nodes on it.
the switch supports fdr10 (40gb) and fdr(56gb). to use fdr you would need a cx3 pro, or a cx4 fdr/edr card. you will also need a mellanox branded cable to support that. this particular switch doesnt unlock that till the firmware is updated to the lastest available, it previously was locked behind a license, but they have since allowed it to operate at 56 for free with a newer firmware.
Last edited:
That's a good price. Alas, they don't ship here...
Same with them.
But I have found a place that does ship here. Shipping costs almost as much as the switch but that still is cheaper than buying it here...
I read on some forum that it isn't easy updating the firmware. Is that really a problem in your experience? Is there maybe a risk to get an older version of the switch that can't be updated to the latest firmware? That would be a bummer.
I went ahead and already bought a bunch of cards and cables (not the Mellanox brand - I'll keep option that for a later time).
And so it begins
Thank you so much for your help in deciding what to buy!!!
Switch firmware wasnt that difficult, id say the hardest part was finding it in the nvidia website:
https://network.nvidia.com/support/firmware/lenovo-archive/
heres the firmware upgrade chart:
Just follow the chart and it should be fine, it is fairly time consuming so i just updated, stepped away for an hour, came back, rebooted, then updated again till i was fully updated.
you will be wanting the PPC version as these are Power PC processors. if you do it local which is what i did you will get a "loading" icon for about 10 minutes before it actually does stuff, so just something to be aware of. its doing what its supposed to, just walk away and let it try its best.
https://network.nvidia.com/support/firmware/lenovo-archive/
heres the firmware upgrade chart:
Just follow the chart and it should be fine, it is fairly time consuming so i just updated, stepped away for an hour, came back, rebooted, then updated again till i was fully updated.
you will be wanting the PPC version as these are Power PC processors. if you do it local which is what i did you will get a "loading" icon for about 10 minutes before it actually does stuff, so just something to be aware of. its doing what its supposed to, just walk away and let it try its best.
Last edited:
Okay, so I have all the stuff ready.
Now the question is, how do I replace the network on my cluster nodes while they are parts of the clusters (PVE and Ceph)? I can't simply add in the Mellanox cards due to a lack of PCIe slots. But if I take out the 10gbe cards and put in the Mellanox cards, there's going to be a disruption of the clusters until I have configures the new cards...
Is there a way to "hibernate" the cluster functions?
Now the question is, how do I replace the network on my cluster nodes while they are parts of the clusters (PVE and Ceph)? I can't simply add in the Mellanox cards due to a lack of PCIe slots. But if I take out the 10gbe cards and put in the Mellanox cards, there's going to be a disruption of the clusters until I have configures the new cards...
Is there a way to "hibernate" the cluster functions?
For small clusters isn't the standard approach to do one node at a time, so the remaining cluster members maintain quorum and things keep on working without interruption?
Last edited:
@jamesthetechie What's the noise level like from those SX6036 switches?
Asking because the older generation Mellanox IB switches used to permanently be in jet engine mode and therefore were super unsuitable for homelab/testlab usage.
Asking because the older generation Mellanox IB switches used to permanently be in jet engine mode and therefore were super unsuitable for homelab/testlab usage.
Well, that 's the problem, isn't?
I have a three node cluster. So I take the first node offline to replace the networking hardware. The remaining two nodes should be able to keep the cluster running. I restart the first node but it won't be connected to the cluster yet. Then I take the second node offline to replace the networking hardware there as well - end of game...
Huh. That's an important question. Should have asked that before I went down that route...@jamesthetechie What's the noise level like from those SX6036 switches?
Asking because the older generation Mellanox IB switches used to permanently be in jet engine mode and therefore were super unsuitable for homelab/testlab usage.
Unfortunately, I can't tell yet, it's still in the box.
As a data point, that's a prime candidate for just wiring directly to each other using those dual port 40GbE adapters. I'm doing that with a small 3 node homelab cluster and it's extremely simple.
Much less complicated than having to add a switch into the mix, let alone muck around with Infiniband again (it's been years).
For mine the switch was quieter than my servers, it was easy enough to have a conversation next to.
One more question about the infiniband cables:
There seem to be (normal) copper cables and there also seem to be fiber optical cables. Both seem to have the same plugs at the end. Can I use either copper or fiber or can I only use copper? And if I can use either, can I mix them in one switch (i.e. connect one PC via copper and connect another PC via fiber)?
Thanks!
There seem to be (normal) copper cables and there also seem to be fiber optical cables. Both seem to have the same plugs at the end. Can I use either copper or fiber or can I only use copper? And if I can use either, can I mix them in one switch (i.e. connect one PC via copper and connect another PC via fiber)?
Thanks!
Yeah, they both work to the same standard (eg "QSFP", "QSFP28", etc). You're fine to plug the fully copper ones (known as "DAC"s) into the exact same switches as the fibre ones as long as you verify that the cable you're using is of the right standard.
The main differences - as far as I know - is that the fibres ones can have very long cables (kilometres with the right gear), whereas the copper ones have a maximum of only a few metres.
The main differences - as far as I know - is that the fibres ones can have very long cables (kilometres with the right gear), whereas the copper ones have a maximum of only a few metres.
i generally recommend the DAC cables as opposed to fibre, their going to draw less power and its going to be a much more simple solution.