Open vSwitch and incorrect RSTP (+ crash on topology change involving Mellanox 10GbE adapter)

[brad_mssw]

Interesting ... good that you got a stack trace, when mine panic'd I didn't.

I did notice that in the trace that it shows NAPI and GRO. In the 4.5 notes, it pretty much says the NAPI system was overhauled:
https://kernelnewbies.org/Linux_4.5#head-5558c630ad32cc1b2c85fb8ab6a4e4f5c0bb64de

It might also be interesting to try disabling GRO with ethtool to see if it works around the problem on 4.4 as well.

 

[kaufenpreis]

Ok, i will try 4.4.30 this late evening first.

 

[gardar]

I've done some further testing, I still have not been able to pull any logs/stack traces in regards to this issue.

I tried various combinations of turning off/on offloading settings with ethtool, but they did not seem to have any affect on this crash.

ethtool -K eth2 gro off
ethtool --offload  eth2  rx off  tx off
ethtool -K eth2 gso off
ethtool -K eth2 rxvlan off
ethtool -K eth2 txvlan off
ethtool -K eth2 sg off
ethtool -K eth2 rxhash off

I can confirm that this issue persists in the following kernels:

Linux pve-sn02-ch01 4.4.35-1-pve #1 SMP Tue Dec 6 09:55:45 CET 2016 x86_64 GNU/Linux
Linux pve-sn01-ch01 4.4.30-040430-lowlatency #201611010007 SMP PREEMPT Tue Nov 1 04:13:22 UTC 2016 x86_64 GNU/Linux
Linux pve-sn04-ch01 4.4.35-040435-lowlatency #201611260431 SMP PREEMPT Sat Nov 26 09:37:07 UTC 2016 x86_64 GNU/Linux

The following kernels don't seem to have this issue:

Linux pve-sn01-ch01 4.5.0-040500-lowlatency #201603140130 SMP PREEMPT Mon Mar 14 05:37:11 UTC 2016 x86_64 GNU/Linux
Linux pve-sn01-ch01 4.7.0-040700-lowlatency #201608021801 SMP PREEMPT Tue Aug 2 22:07:01 UTC 2016 x86_64 GNU/Linux

All of the kernels, the faulty and working ones have the same kernel module version for mlx4. 2.2-1 (Feb 2014). I did also try with the OFED version 3.4-1.0 and got the same results.

I checked the ethtool default settings on all of those kernels, you can see them here: https://notehub.org/wqbov

One thing I noticed is that the busy-poll is "on" on the kernels that I'm having issues with, but the working kernels both have busy-poll set to "off".
I'm not sure if that's related or not.

@fabian any further testing I can do?

 

[fabian]

there have been quite some (tree-wide) changes regarding the busy-pool stuff in 4.5 - I don't think we can backport all of those.. I can try to build a 4.4 kernel with disabled CONFIG_NET_RX_BUSY_POLL for testing..

 

[gardar]

there have been quite some (tree-wide) changes regarding the busy-pool stuff in 4.5 - I don't think we can backport all of those.. I can try to build a 4.4 kernel with disabled CONFIG_NET_RX_BUSY_POLL for testing..

A test build would be good, I was underway of building it myself.

Are there any plans of supporting 4.5+ in the forseeable future?

 

[gardar]

Compiled the kernel with

-d CONFIG_NET_RX_BUSY_POLL

in the Makefile for the proxmox kernel, but it seems like it did not disable busy-poll

 

[brad_mssw]

My guess is proxmox prefers LTS kernels. 4.9 was supposed to be announced as LTS, which was just released Monday, but I haven't seen any confirmation of its LTS status, so it might have been pushed to 4.10.

 

[gardar]

@fabian Any ETA on that kernel? I'll be moving those servers in to production soon, so I'm running out of time to do further testing.

@brad_mssw Yes that's what I'm assuming too and it's quite understandable, that's why it would be great if there is a solution/workaround to this issue we are having. I'd prefer to be able to use a official Proxmox kernel.

 

[fabian]

@fabian Any ETA on that kernel? I'll be moving those servers in to production soon, so I'm running out of time to do further testing.

sorry for the delay.. seems like it's not really possible on 4.4 without backporting a whole slew of changes. if you manage to either convince the upstream LTS maintainer or the ubuntu kernel team of backporting the relevant changes, we will include them - otherwise I don't think we will carry them on our own..

 

[brad_mssw]

Those that have it working, what network cards are you using?

In my test lab, I have it working with 2x igb and 2x ixgbe ports and it seems to work well.

However I just set up a new cluster and it uses 2x igb, then 2x the newer Intel X710/XL710 which uses the i40e driver and it clearly doesn't work, when I interconnect the 10GbE cables, at least one node (of 3) will stop pinging, but this doesn't kernel panic, it recovers once the cables are unplugged again. Nothing is really logged when this occurs, but I didn't have an active console to query OVS to see what it thought was going on.

Regardless since this works on one system with the exact same configuration (identical to PVE wiki for OVS), kernel (using mainline 4.9 generic from ubuntu ppa), and latest pve-no-subscription packages ... but this system fails, the differentiating factor appears to be the NIC driver. So I'd like to know what others are using.

I just ordered a set of Mellanox Connect X-4Lx cards to be here next week to see if those work. Seems like a lot of people prefer Mellanox, just hope I picked the "right" generation card.

 

[brad_mssw]

The Mellanox Connect X-4Lx cards worked great with the mlx5 driver.

Everyone: avoid Intel i40e, even if using their newer 10GbE X710 cards (i40e isn't specific to 40GbE). The older 10GbE generation that used the ixgbe appears to be fine, so the i40e driver is borked.

 

[brad_mssw]

@fabian Looks like 4.9 is officially LTS now, any chance the PVE kernel will be updated to this in the near-ish future?

 

[tom]

@fabian Looks like 4.9 is officially LTS now, any chance the PVE kernel will be updated to this in the near-ish future?

We try to get 4.9 soon, but as a new kernel branch needs intensive testing you cannot expect this immediately.