Need 10GBe adapter recommendations

Jul 18, 2018
14
0
6
40
I've been having a time, not a good time, but a time.

When I can back from vacation I would my Nortel 5520 had died. Much despair and weeping.

After some research I decided I would go with a Mikrotik CRS328 so I could use the sweet sweet 10GBe links I alread had installed in the server and my PC.

The mikrotik came in and it was a slow motion disaster. The 10GB trunk did not work as expected, and there was more wailing and despair.

After a weekend of aggravation, I removed the fiber trunk and went back to a copper trunk, and everything has been steady since Sunday morning.

If proxmox is configured to autostart the bridge ties to my mellanox Connectx-2, the service wont start. This lead me to look for a new driver and I'm finding all kinds of things about those being way too old and having weird vlan issues.

I got the networking service to restart by turning off VLAN aware in the bridge. So the link will probably work, but not as a trunk. More despair.

I think it's somewhat clear that I need a new 10gbe adapter for my server if I want to rock a 10GB link to the switch. Do you guys have recommendations that are absolutely gonna work? One nice fella in the homelab discord mentioned Intel X520-DA2.

Hardware used

  • Dell T6230 running Proxmox 5.2-5
    • Mellanox Connectx-2
      • Finisar FTLX8571D3BCL
  • CRS328-24P-4S+RM running SwOS 2.8
    • Finisar FTLX8571D3BCL
 
Hi,

Intel is the most used and so the best-tested nics.
Also, Mellanox and Broadcom are ok, but I would prefer Mellanox over Broadcom.
 
I don't recommend Mellanox. I've got Connectx-3 on my cluster and can't pass VLANID above 126 to VMs on bridged interface.
What do you mean? I have Connectx-2 in ethernet mode and the bridge seems to be working fine (including vlanid).
 
What do you mean? I have Connectx-2 in ethernet mode and the bridge seems to be working fine (including vlanid).

I have so many questions!

  1. What version of proxmox?
  2. What driver are you using for card?
  3. What firmware is on the card?
  4. Which Vlans are you using on your trunk?
 
What do you mean? I have Connectx-2 in ethernet mode and the bridge seems to be working fine (including vlanid).

connect-x3 is limited to 128 vlans max. (I have verified this with mellanox support). But I can setup vlan 300, 400, 4000 without problem.
only vlan aware bridge don't work well, because proxmox setup a trunk with vlan 2-4096 by default, so it's more than the 128 max total vlans.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!