Looking for a "cheap" way to upgrade to 40gibt
- Thread starter proxwolfe
- Start date
Dual ConnectX-3 cards are stupid cheap and support 40gb ethernet or infiniband depending on what they are currently flash to. The quad port cards are pretty much x4 the price as the dual port cards, so really depends on the price per port you are looking to spend. The expensive part of the whole thing is going to be switch. If you got a small handful of servers, you can skip the switch and go direct attached from one to another.
I do not know of any NIC cards that support RJ45 over 10gb. You can use QSFP+ cards with Transceiver and Cat8e cables to do 40gb RJ45, but I am pretty sure that will be more expense than just using QSFP+ cables.
I do not know of any NIC cards that support RJ45 over 10gb. You can use QSFP+ cards with Transceiver and Cat8e cables to do 40gb RJ45, but I am pretty sure that will be more expense than just using QSFP+ cables.
Thank you for this suggestion. These cards are indeed ridiculously cheap. I shall investigate this further. My first impression is that Infiniband is the better technology - would you agree?
I do, in fact only have a small home lab cluster of three nodes. But I am a bit reluctant to wire them directly without a switch because I fear that I would mess up the configuration. But I might change my mind when I see the price tag of a switch...
I do, in fact only have a small home lab cluster of three nodes. But I am a bit reluctant to wire them directly without a switch because I fear that I would mess up the configuration. But I might change my mind when I see the price tag of a switch...
To chime in, i do recommend running infiniband, you can use IPoIB which will basically act as a normal network card.
the cx3 cards that were recommended do work pretty good and are usually around $20 per for a dual port.
you can get a sx6036 infiniband switch for dirt cheap, around $100-$150 and i would highly recommend one for a home lab.
there are some slight issues being that you will need to update the firmware to the latest which can take some effort, and you will need mellanox branded IB cables, but other than that its a fairly plug and play solution.
RDMA alone is a good reason to get IB.
the cx3 cards that were recommended do work pretty good and are usually around $20 per for a dual port.
you can get a sx6036 infiniband switch for dirt cheap, around $100-$150 and i would highly recommend one for a home lab.
there are some slight issues being that you will need to update the firmware to the latest which can take some effort, and you will need mellanox branded IB cables, but other than that its a fairly plug and play solution.
RDMA alone is a good reason to get IB.
Is bandwidth fever a thing? Because I think I'm feeling it 
But I also found sx6015 and sx6025 that are quite affordable. Main difference, apart from the number of ports on the sx6015, seem to be managed vs unmanaged. Would I need the managed version when I want to partition the switch into separate networks? I currently have three networks (Proxmox cluster, Ceph cluster and PBS) and I wouldn't want to set up three switches.
When googling for the differences, I happened upon a discussion about used switches and someone pointed out that they may come without ethernet license. Is that something I would need to look out for, if I want to use (only) IPoIB (or is that in relation to using the switch as an ethernet switch itself)?
A concern is that these switches come with 2 300W PSUs. Would you be able to give me a rough estimate of the power consumption?
And lastly, some of this equipment is described as "40gbe", some is "40gbe/56gbe" and some is "56gbe" - is that all the same (possibly depending on the quality/standard of the cables) or are these different (incompatible) standards and I need to make sure to get all parts with the same designation?
Thanks!
With a quick check on ebay, I found a couple sx3036 but, alas, at substantially higher prices. Probably need to watch that space and strike in a good moment...
But I also found sx6015 and sx6025 that are quite affordable. Main difference, apart from the number of ports on the sx6015, seem to be managed vs unmanaged. Would I need the managed version when I want to partition the switch into separate networks? I currently have three networks (Proxmox cluster, Ceph cluster and PBS) and I wouldn't want to set up three switches.
When googling for the differences, I happened upon a discussion about used switches and someone pointed out that they may come without ethernet license. Is that something I would need to look out for, if I want to use (only) IPoIB (or is that in relation to using the switch as an ethernet switch itself)?
A concern is that these switches come with 2 300W PSUs. Would you be able to give me a rough estimate of the power consumption?
And lastly, some of this equipment is described as "40gbe", some is "40gbe/56gbe" and some is "56gbe" - is that all the same (possibly depending on the quality/standard of the cables) or are these different (incompatible) standards and I need to make sure to get all parts with the same designation?
Thanks!
heres an amazon listing for the sx6036, i bought 2 from this place and they worked great:
Mellanox sx6036
heres one for ebay thats cheaper:
Mellanox sx6036
you would for sure want a managed one, infiniband needs a subnet manager and they come installed on managed switches. if you dont have a managed switch then you would need to install and configure opensm on your hosts and its not worth the difference in price. you can set up separate partitions in the managed switch.
the IB switches do not come with ethernet capability by default and only do infiniband out of box. you can get an ethernet license but i havnt found it to be all that helpful and i just instead use IPoIB which allows me to use IB how i want it as well as standard ethernet traffic without having to buy a license.
the switch consumes about 50w with 4 nodes on it.
the switch supports fdr10 (40gb) and fdr(56gb). to use fdr you would need a cx3 pro, or a cx4 fdr/edr card. you will also need a mellanox branded cable to support that. this particular switch doesnt unlock that till the firmware is updated to the lastest available, it previously was locked behind a license, but they have since allowed it to operate at 56 for free with a newer firmware.
Mellanox sx6036
heres one for ebay thats cheaper:
Mellanox sx6036
you would for sure want a managed one, infiniband needs a subnet manager and they come installed on managed switches. if you dont have a managed switch then you would need to install and configure opensm on your hosts and its not worth the difference in price. you can set up separate partitions in the managed switch.
the IB switches do not come with ethernet capability by default and only do infiniband out of box. you can get an ethernet license but i havnt found it to be all that helpful and i just instead use IPoIB which allows me to use IB how i want it as well as standard ethernet traffic without having to buy a license.
the switch consumes about 50w with 4 nodes on it.
the switch supports fdr10 (40gb) and fdr(56gb). to use fdr you would need a cx3 pro, or a cx4 fdr/edr card. you will also need a mellanox branded cable to support that. this particular switch doesnt unlock that till the firmware is updated to the lastest available, it previously was locked behind a license, but they have since allowed it to operate at 56 for free with a newer firmware.
Last edited:
That's a good price. Alas, they don't ship here...
Same with them.
But I have found a place that does ship here. Shipping costs almost as much as the switch but that still is cheaper than buying it here...
I read on some forum that it isn't easy updating the firmware. Is that really a problem in your experience? Is there maybe a risk to get an older version of the switch that can't be updated to the latest firmware? That would be a bummer.
I went ahead and already bought a bunch of cards and cables (not the Mellanox brand - I'll keep option that for a later time).
And so it begins
Thank you so much for your help in deciding what to buy!!!
Switch firmware wasnt that difficult, id say the hardest part was finding it in the nvidia website:
https://network.nvidia.com/support/firmware/lenovo-archive/
heres the firmware upgrade chart:
Just follow the chart and it should be fine, it is fairly time consuming so i just updated, stepped away for an hour, came back, rebooted, then updated again till i was fully updated.
you will be wanting the PPC version as these are Power PC processors. if you do it local which is what i did you will get a "loading" icon for about 10 minutes before it actually does stuff, so just something to be aware of. its doing what its supposed to, just walk away and let it try its best.
https://network.nvidia.com/support/firmware/lenovo-archive/
heres the firmware upgrade chart:
Just follow the chart and it should be fine, it is fairly time consuming so i just updated, stepped away for an hour, came back, rebooted, then updated again till i was fully updated.
you will be wanting the PPC version as these are Power PC processors. if you do it local which is what i did you will get a "loading" icon for about 10 minutes before it actually does stuff, so just something to be aware of. its doing what its supposed to, just walk away and let it try its best.
Last edited: