IBM 49Y7970/49Y7972 X540-DA2 (2 Port 10GBASE-T)

[stuartbh]

ProxMox users, developers, et alia:

Subject to this posting is a model of 10GB pcie card that I am considering to obtain for a couple of my x3650 servers that are running ProxMox. My initial concern is what if any considerations (with respect to ProxMox, firmware, drivers, etc...) ought I consider in contemplating using those cards? Do they seem an intelligent choice for use in a ProxMox environment?

My initial configuration for testing will be to use a couple of cross over cables betwixt the ports on the two servers, though I eventually am going to get a 10GB switch, I am just in a proof of concept phase first.

Ideas?


Thanks in advance!

Stuart

 

[meyergru]

I once made the mistake to use 10 GbE adapters before I realised some things and replaced them with SFP+ variants. I suspect you are now in the same spot.

Therefore, just a few facts:

1. X540-DAx are SFP+ cards, X540-Tx are 10 GbE. From what you write about cross-over cables (which are not even needed because of Auto-MDIX), it seems you refer to 10 GbE variants, which would be X540-T2 (or T1 with one port). In order to use 10 GbE with an X540-DAx, you would need additional 10 GbE SFP+ modules which are not cheap.

2. 10 GbE tends to have high power usage (~ 2 Watts per port per side). Thus, if your servers or switches are not too far away, I would use DAC cables. They are cheap and use much less power than a 10 GbE connection. Many switches have SFP+ slots as well.

3. The Intel X540-Tx cards are old style cards that can only handle 1 and 10 GBit, which is less than ideal, because even if you need more than 5m of length, more often than not, you will find than 10 GbE does not work over older cabling, whereas 2.5 or 5 GbE does. Matter-of-fact, X550-Tx adapters can handle those lower speeds. Also, there are many mainboards and devices which already bring 2.5. GbE to the table.

Thus, it depends on what your constraints are: If the servers and your switch are in the same room, I would use X540-DAx with DAC cables as a cheap solution (I actually do). If they are not, you are probably better off with X550-Tx adapters, which handle intermediate speeds and use less power than X540-Tx NICs. There are other options available with different chipsets, which may be cheaper.

 

[stuartbh]

Meyergru,

Thank you for your brisk response. I am a bit new to using network cards with SFP+ or using 10Gb networking in general, so I am happy to learn from others instead of repeating the mistakes of others (hence why I posted here prior to acquiring any hardware).

A friend always tells me, "A wise man can learn from a fool but a fool can learn from no one." to which I always counter: "Given that is the case, it is compellingly interesting how many fools have wise men as students!" Make no mistake, I am not intending to imply you are a fool!!

I once made the mistake to use 10 GbE adapters before I realised some things and replaced them with SFP+ variants. I suspect you are now in the same spot.

Therefore, just a few facts:

1. X540-DAx are SFP+ cards, X540-Tx are 10 GbE. From what you write about cross-over cables (which are not even needed because of Auto-MDIX), it seems you refer to 10 GbE variants, which would be X540-T2 (or T1 with one port). In order to use 10 GbE with an X540-DAx, you would need additional 10 GbE SFP+ modules which are not cheap.

Well there is surely a far greater level of alternative configuration options going with the SFP+ cards over the 10Gb Ethernet cards and there also seems to be almost zero price differential (at least with older cards), thus I stand convinced to use SFP+ at this juncture. I do admit I forgot to even look if Auto-MDIX was applicable to 10GbE cards (which I presume it is now from your comments) though irrelevant at this juncture since I will go with SFP+ cards.

I also see that I made a mistake in referring to the 10GbE card as a "X540-DA2" in the subject line of my initial posting, as it should have read "X540-T2". Thank you for pointing that out.

2. 10 GbE tends to have high power usage (~ 2 Watts per port per side). Thus, if your servers or switches are not too far away, I would use DAC cables. They are cheap and use much less power than a 10 GbE connection. Many switches have SFP+ slots as well.

All the switches I have looked at thus far that have 10Gb SFP+ ports (some had 10GbE ports too) so that is absolutely correct as a presumption. Worthy of notation is that I was unaware of the power consumption difference on 10GbE, that is surely an interesting factor for consideration as well since this is for a home lab, not a data center deployment.

I am currently planning to use 3 servers that are 2U each all racked one above the other. Thus, the furthest would be the top and bottom SFP+ ports being 3-4U in distance from each other. As such, it would be no problem to keep them that close in my rack persistently and leveraging DAC cables would not be a problem from that perspective.

3. The Intel X540-Tx cards are old style cards that can only handle 1 and 10 GBit, which is less than ideal, because even if you need more than 5m of length, more often than not, you will find than 10 GbE does not work over older cabling, whereas 2.5 or 5 GbE does. Matter-of-fact, X550-Tx adapters can handle those lower speeds. Also, there are many mainboards and devices which already bring 2.5. GbE to the table.

I do not think I will need more than 5m of length; though purchasing and using CAT7 cable is not a huge deal either. For foregoing notwithstanding, the flexibility of using SFP+ cards does seem to be a far more superior choice and a justifiable reason for to dispense with considering the X540-Tx based cards.

2.5GbE is interesting, though I have no hardware that has multigig or 2.5GbE ports in them yet. I have one consumer router (I do not use it currently, it was a gift from my ISP) that has an SFP+ port on it (I think it is SFP+ not SFP), nor do I know if the firmware allows its configuration or usage. It will be interesting to see what direction consumer routers go in the near future. For now, it is not something I am too worried about. My prognostication is that multigig chipsets will be inexpensive enough that multigig ports in consumer routers will end up the way ahead (supporting 2.5GbE, 5GbE, and 10GbE).

Thus, it depends on what your constraints are: If the servers and your switch are in the same room, I would use X540-DAx with DAC cables as a cheap solution (I actually do). If they are not, you are probably better off with X550-Tx adapters, which handle intermediate speeds and use less power than X540-Tx NICs. There are other options available with different chipsets, which may be cheaper.

Using X540-DA2 cards with either DAC cables seems a good idea. In fact, I had the idea of using DAC cables for direct connections (absent a switch) but now realize that DAC cables could also get plugged into a switch too (something I had not pondered before), so it offers me a level of future expansion once I select a 10Gb switch to obtain.

Thus, at this point I think I am going to look at acquiring X540-DA2 cards and DAC cables for them, which would let me either directly connect ports on one X540-DA2 to another X540-DA2 initially, leaving me with the ability in the future to plug the DAC cable into a switch once I acquire one and get it racked.

The one concern I do have is relative to using a card like an X540-DA2 in a ProxMox environment? Are there any concerns software wise or firmware wise with doing this?

Thanks again in advance for your time and consideration regarding the instant matter before us.

Stuart

 

[meyergru]

As I wrote, this works fine OOTB, although I used white-label Intel variants from Aliexpress. The Intel-based cards have a wide ranging support even when you look at FreeBSD-based routers like pfSense or OpnSense, even more for Linux-based solutions like Proxmox. I use both with switches employing SFP+ slots and DAC cables. I also have a NAS with an X540-DA1.

Many modern devices support 2.5 GbE already, many mainboards with either Intel 225 or 226 or RealTek 8125 NICs, even cheap N5105 routers have that. Also, there are nice solutions for switches like the Unifi Enterprise 24 POE or the Unifi USW Aggregation, although cheaper solutions exist.
I control my Unifi switches and APs with a Unifi controller running under Proxmox.

I have found that DAC cabling in the server room works fine and the 2.5 GbE POE ports work even for older house cabling with CAT5 without limiting the speed too much even for potent PCs. 2.5 GbE uses almost no more power than 1 GbE. In my server room, I have saved ~17 Watts by replacing the 10 GbE links with DAC, because DAC cables use virtually no power whereas my 4 10 GbE connections used 20 Watts total.

 

[stuartbh]

As I wrote, this works fine OOTB, although I used white-label Intel variants from Aliexpress. The Intel-based cards have a wide ranging support even when you look at FreeBSD-based routers like pfSense or OpnSense, even more for Linux-based solutions like Proxmox. I use both with switches employing SFP+ slots and DAC cables. I also have a NAS with an X540-DA1.

Well, as far as cards go I would prefer to get the real IBM versions that easily can have their firmware updated when I do full system firmware upgrades on the x3650 systems and that and it has good parts on it. I have heard far too many stories of these Aliexpress clones burning out in short order because they are made with counterfeit chip sets and the like. No thank you. The IBM original cards with an SFP+ (fiber with LC connector seems to be common) can be had for about $30 a unit these days, no reason to use Chinese junk in my view.

That said, I run ProxMox and then run any thing (like pfSense for example, which I use) under ProxMox, so all I need is for ProxMox to support the card and a bridge interface goes to the VM.

I plan to use switches with SFTP+ and DAC cables as well. My NAS (currently) is a TrueNAS SCALE server running an an old x3650 M1 (7970 officially) with 48GB of RAM.

Many modern devices support 2.5 GbE already, many mainboards with either Intel 225 or 226 or RealTek 8125 NICs, even cheap N5105 routers have that. Also, there are nice solutions for switches like the Unifi Enterprise 24 POE or the Unifi USW Aggregation, although cheaper solutions exist.
I control my Unifi switches and APs with a Unifi controller running under Proxmox.

That may well be true (including newer routers) but my hardware is older and stands absent these newer chipsets like the i225 and i226 or RealTek 8125 and such. Some of the Chinese N5101 mini PCs have the i226 in them with 4 ports or the like. I might get one in the future but do not need it as yet.

I have heard good things about Unifi but have yet to learn more about their hardware.

I have found that DAC cabling in the server room works fine and the 2.5 GbE POE ports work even for older house cabling with CAT5 without limiting the speed too much even for potent PCs. 2.5 GbE uses almost no more power than 1 GbE. In my server room, I have saved ~17 Watts by replacing the 10 GbE links with DAC, because DAC cables use virtually no power whereas my 4 10 GbE connections used 20 Watts total.

I think my cabling is either CAT5E or CAT6? I'd have to check, though I probably have some CAT5 for sure. My network is not really overrun at 1GbE but 2.5GbE will likely find its way onto my network as newer hardware I obtain has it.

Thank you again for all your thoughts and comments as they are going to impact the choices I will make in the near future hardware wise.

Stuart