Recommendation - Quad Port Gigabit PCIe Ethernet card?

[stuartbh]

Forum members,

I am interested in selecting a quad port Ethernet card to be used with ProxMox and am wondering what people's experiences have been and what opinions are out there.

One card I was looking at is the i350-T4v2 but I am curious what others think? For what it is worth the card would be going into some Xeon type older servers I use.

I know this question has been asked before but then also so to does BIOS and driver support change, chipsets, etc...

Thanks in advance.

Stuart

 

[guletz]

Hi,

Maybe you can try to explain why you consider that you need a 4 x Gbit port interface .... maybe you are on the wrong way

Good luck / Bafta !

 

[stuartbh]

Bafta,

It's funny I lived in Southern California for a long time around all these movie/entertainment type people so initially when I saw Bafta in your signature line I thought British Academy of Film, Television, and Arts (then of course I realized it was your name). I never even knew of that organization until I lived around the movie people there.

It is not that my use cases are so secret or that there might not be some worthy technical discussion about them, but I guess I am just curious what difference it makes? I am asking for a list of what quad port Ethernet cards are considered compatible with the current crop of available drives in ProxMox and that can work with reasonable efficiently whilst being available (hopefully) in the secondary market or at a reasonable price for a home lab user.

Thanks!

Stuart

 

[Dunuin]

i350-T4v1 works absolutely fine here. It's just missing the SR-IOV of the v2 version, which would be nice to have. But make sure not to buy a fake one, as a big portion of them out there is not genuine: https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/

 

[stuartbh]

i350-T4v1 works absolutely fine here. It's just missing the SR-IOV of the v2 version, which would be nice to have. But make sure not to buy a fake one, as a big portion of them out there is not genuine: https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/

Dunuin,

How important or useful is SR-IOV if I may ask? I am being over the top about this feature or can it be forgone and there would be little impact for a home lab user?

I am well aware of the issue with respect to counterfeit cards and as such would only buy one from one of the many eBay sellers that deal in used servers / associative parts. etc... and is located in North America. In this case, it is perhaps advantageous to know that the seller is stating it is definitely not new!

Does the manufacturer matter at all (IBM vs Dell vs Cisco) as long as it is an i350-T4? I presume most will be v1 at this point.

Stuart

 

[Dunuin]

SR-IOV will allow you to split your physical NIC into multiple virtual NICs. Lets say you got 16 VMs and you want to use PCI passhrough for better performance and/or hardware offloading. Without it you can only passthrough the physical ports, so that NIC can only serve 4 VMs. With SR-IOV you could split it into 16 virtual NICs, so you got 16 functions, each with its own IOMMU group so all 16 VMs can use a passthroughed NIC.

So can be very useful in a hypervisor. But it's more optional, as you can also use virtual NICs on the software level. And virtual virtio NICs are fast enough to not be the bottleneck with Gbit NICs. SR-OV would be way more important when using 10Gbit or even faster NICs.

 

[stuartbh]

SR-IOV will allow you to split your physical NIC into multiple virtual NICs. Lets say you got 16 VMs and you want to use PCI passhrough for better performance and/or hardware offloading. Without it you can only passthrough the physical ports, so that NIC can only serve 4 VMs. With SR-IOV you could split it into 16 virtual NICs, so you got 16 functions, each with its own IOMMU group so all 16 VMs can use a passthroughed NIC.

So can be very useful in a hypervisor. But it's more optional, as you can also use virtual NICs on the software level. And virtual virtio NICs are fast enough to not be the bottleneck with Gbit NICs. SR-OV would be way more important when using 10Gbit or even faster NICs.

Dunuin,

So if I can get a i350-T4v2 at a good price, then it seems it is a worthy investment. That said, when you say "virtual virtio NICs", I presume you mean using the virtio driver with a NIC that has is accessed via vmbr0 or the like port, or is there some special virtual virtio NIC configuration to be applied?

Presuming I did hypothetically acquire an i350-T4v2, what steps would I take to leverage the 16 virtual functions? I have setup PCI passthrough before (been a while, but I did do it), how would this be any different? If I do use these 16 virtual functions does that mean each VM gets a "virtual 4 port card with 4 ports virtualized per VM"?

Stuart

 

[Dunuin]

So if I can get a i350-T4v2 at a good price, then it seems it is a worthy investment. That said, when you say "virtual virtio NICs", I presume you mean using the virtio driver with a NIC that has is accessed via vmbr0 or the like port, or is there some special virtual virtio NIC configuration to be applied?

Jup, I mean that.
Here is a nice video showing the difference between using SR-IOV virtual functions and using only the physical function with a bridge on the hypervisor: https://www.intel.com/content/dam/s...thernet-products/Intel-SR-IOV-Explanation.mp4

Presuming I did hypothetically acquire an i350-T4v2, what steps would I take to leverage the 16 virtual functions?

First your mainboards needs to support SR-IOV and you need to enable it in BIOS. Then you need to tell the NICs firmware to use it. In addition to the 4 physical functions PVE should then see 16 additional virtual functions you can passthrough into a VM like you got 16 additional NICs.

I have setup PCI passthrough before (been a while, but I did do it), how would this be any different? If I do use these 16 virtual functions does that mean each VM gets a "virtual 4 port card with 4 ports virtualized per VM"?

Not sure how the virtual functions will be assigned to the 4 physical ports, as all my SR-IOV capable NICs are just single port. I would guess you can define that somewhere in the firmware or driver.

 

[guletz]

One card I was looking at is the i350-T4v2 but I am curious what others think?

Details
Product code	RB44Ge
10/100/1000 Ethernet ports	4
Can be used indoors	Yes
Chipset	Atheros AR8131/M
Format	PCIe 4X

 

[stuartbh]

Jup, I mean that.
Here is a nice video showing the difference between using SR-IOV virtual functions and using only the physical function with a bridge on the hypervisor: https://www.intel.com/content/dam/s...thernet-products/Intel-SR-IOV-Explanation.mp4

This video presentation was rather instructive in no uncertain terms. I appreciated that the volume of the video was sufficient to use my internal speakers as well as the clear English and how comprehensive the disquisition was.

First your mainboards needs to support SR-IOV and you need to enable it in BIOS. Then you need to tell the NICs firmware to use it. In addition to the 4 physical functions PVE should then see 16 additional virtual functions you can passthrough into a VM like you got 16 additional NICs.

This is probably the most valuable piece of information for me, as my servers are a decade or so old (x5460 and x5580 xeon based systems) and their BIOS has no such option that I could see concerning SR-IOV. If I am understanding you correctly, this is a show stopper right here and leads me to believe that for these systems a card supporting SR-IOV would never function using the SR-IOV technology due to lacking BIOS support. That said, it is worthy of notation that these systems do boot via UEFI, I wonder if support can be added via UEFI?

Not sure how the virtual functions will be assigned to the 4 physical ports, as all my SR-IOV capable NICs are just single port. I would guess you can define that somewhere in the firmware or driver.

Well, I imagine I'd need to do more research on this, though, as was already stated, absent the BIOS support the v2 cards hardly seem worth it for the systems I am considering them for currently. I will likely grab a v1 version of the card based on your prior comments regarding the requisite of motherboard BIOS support.

 

[stuartbh]

Details
Product code	RB44Ge
10/100/1000 Ethernet ports	4
Can be used indoors	Yes
Chipset	Atheros AR8131/M
Format	PCIe 4X

Thank you for the recommendation, I will research this card as well.