Initial switch interface does not work

[ninemate]

Hi all,

I seem to have found an interesting issue, happened to me 2 out of 2 tries so it can be replicated.
Hardware I'm using:

• Dell PowerEdge R750 with Broadcom NetXtreme Gigabit Ethernet NIC's (tested with the firmware it shipped with and the newest from Dell)
• Juniper EX4300-48T switch

Steps to replicate the issue:

1. Configure an access port and a bond interface on the switch (the plan is to have 2 bonds, one optics and one UTP, but to download OpenvSwitch we need a working interface).
   Disable unneccessary interfaces.
2. Attach proxmox install image, go through the process. Choose the interface that has link as the management interface. Proxmox sets up /etc/network/interfaces with a vmbr0 as it should do, but no matter what you do traffic won't work.

Using tcpdump you can see traffic, ARP requests mainly and some virtual router protocol packets. Interesting thing is that the proxmox host refuses to reply to ARP, as if it was mute. Ping obviously acts like you'd think, 'Destination host unreachable'.

'ip neighbor' acts strange as well, IP's I manually add (nameservers from resolv.conf, hosts I try to ping) are displayed with or without mac, marked as STALE or FAILED. IPv6 hosts are found automatically - the way it should be with IPv4. if I disable IPv6 they disappear but that's it.

The only fix I found was to plug the server into another switchport - with a 100% identical config I might add. Somehow the interface used while installing proxmox / setting up the server does not seem to work afterwards. It doesn't even work while install, the server should get a DHCP address but defaults to the hardcoded ones. Plugging it into another port after completing the install does the trick but plug it back to the initial one and it stops working again. On the switch side everything seems to be in order, link speed is communicated, but no mac on the port. It is set to learn, but the host does not communicate it's mac for some reason. I'm sure it's not the switch, any other machine works just fine.

I've found this thread but it does not seem to help my case. Might it be that we've chosen an unlucky combo of NIC, linux kernel, switch and driver? Anything I forgot feel free to ask.

Thanks and take care,
Máté

 

[ninemate]

So, I've tried this with multiple Dell servers ranging from R710 to R750, different Broadcom NICs. Each and every one of those did the same. The two ways to get networking on them is:

• Only attach cables after PVE has been installed.
• Plug them into a different switchport.

as with tcpdump, running it with "-i any" and no port restriction just causes the kernel to essentially drop 98% of packets.

anyone encounter this? my journeys to the server room are getting pretty damn boring and honestly, debugging this is just infuriating.

M