VLAN with tag above 126 problem

[spirit]

response from mellanox support:

"
Based on the information provided, our engineers came back with the following explanation:

"With the older kernels, the driver was never able to report the error regarding the limitation, this was resolved in later kernels. Still, even being able to configure > 128 vlan's, the limitation for ConnectX-3 is still 128 vlan's"

Please update the case if you need any further assistance regarding this request or we can proceed with the closing of the case.
"

:/

 

[mir]

Not only restricted in number of vlans but also restricted in the vlan numbering. If they have to choose a byte to store vlans why choose a signed byte! There is no such thing as a negative vlan number.

 

[Stoiko Ivanov]

Ok - learned something new - w.r.t. limitation of drivers ... :/

just out of curiosity: does disabling the offloading features change anything (if someone has time/is interested)/make vlans above 126 configurable?
(ethtool -k $nic - to list the available ones)

 

[raku]

Bumping

@Stoiko Ivanov - any specific offloading feature?

I've disabled anything I was able:

ethtool -k ens15d1 |grep offload
Cannot get device udp-fragmentation-offload settings: Operation not supported
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off

Still not able to configure vlan over 126.

Looks like I have to get used to vlan mapping on the switch :-/

 

[Stoiko Ivanov]

I thought about the rx-vlan/tx-vlan -offload settings primarily (helped me with a different NIC once...).
If you still want to experiment - you could take a look at the other features (i.e. without the |grep offload) - AFAIR the offered flags vary from driver-version (thus kernel version) to version, and also change with newer versions of ethtool.

 

[raku]

I've disabled anything with vlan in name:

ethtool -k ens15d1 |grep vlan
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-vlan-offload: off
tx-vlan-offload: off
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off [requested on]
rx-vlan-stag-filter: on [fixed]

No luck.

Here's everything what ethtool returns:

ethtool -k ens15d1
Features for ens15d1:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
   tx-checksum-ipv4: on
   tx-checksum-ip-generic: off [fixed]
   tx-checksum-ipv6: on
   tx-checksum-fcoe-crc: off [fixed]
   tx-checksum-sctp: off [fixed]
scatter-gather: on
   tx-scatter-gather: on
   tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
   tx-tcp-segmentation: on
   tx-tcp-ecn-segmentation: off [fixed]
   tx-tcp-mangleid-segmentation: off
   tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off [requested on]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on

 

[mkunin]

Hi,

I have ProxMox (pve-manager/4.4-13/7ea56165 (running kernel: 4.4.44-1-pve)) with Mellanox ConnectX-3Pro NIC (MCX312B-XCCT). The firmware is 2.35.5100 , and in kernel driver driver: mlx4_en version: 2.2-1 (Feb 2014). I don't have any issue with VLANs.

I am installing the ProxMox 5.2 on other server with 2.42.5000 and there it is not working.

 

[mkunin]

So I changed the configuration of the bridge:
auto vmbr0
iface vmbr0 inet manual
bridge-ports enp3s0d1
bridge-stp off
bridge-fd 0

And restart the server. Now it is working with ProxMox 5 and 2.42.5000 firmware and in kernel 2.2 driver.

 

[Stoiko Ivanov]

@mkunin: what exactly did you change? Do you mean the perdictable network interface names (enp3s0d1 instead of eth0)?

 

[raku]

I think he removed:

   bridge-vlan-aware yes
   bridge-vids 2-4094

 

[spirit]

I think he removed:

   bridge-vlan-aware yes
   bridge-vids 2-4094

the problem with last mellanox driver, with bridge vlan aware, is that vlans are allocated from 2 to 4096 directly,
and it's blocking around 126-128.

without bridge vlan aware, you can allocated the vlan you want (but still limited to 128 total vlan numbers)

 

[mkunin]

@mkunin: what exactly did you change? Do you mean the perdictable network interface names (enp3s0d1 instead of eth0)?

I removed:
bridge-vlan-aware yes
bridge-vids 2-4094

 

[tobias1]

I got to this thread googling "proxmox vlan 126" after determining that a new vlan I was adding (also in the 500 range, coincidentally) was not passing traffic between the nic and proxmox bridge, and stopped at 126:

# bridge vlan show vmbr0
port              vlan-id 
<snip>
enp1s0            1 PVID Egress Untagged
                  2
                  3
                  4
                  5
                  6
                  7
                  8
                  9
                  10
                  11
                  12
                  13
                  14
                  15
                  16
                  17
                  18
                  19
                  20
                  21
                  22
                  23
                  24
                  25
                  26
                  27
                  28
                  29
                  30
                  31
                  32
                  33
                  34
                  35
                  36
                  37
                  38
                  39
                  40
                  41
                  42
                  43
                  44
                  45
                  46
                  47
                  48
                  49
                  50
                  51
                  52
                  53
                  54
                  55
                  56
                  57
                  58
                  59
                  60
                  61
                  62
                  63
                  64
                  65
                  66
                  67
                  68
                  69
                  70
                  71
                  72
                  73
                  74
                  75
                  76
                  77
                  78
                  79
                  80
                  81
                  82
                  83
                  84
                  85
                  86
                  87
                  88
                  89
                  90
                  91
                  92
                  93
                  94
                  95
                  96
                  97
                  98
                  99
                  100
                  101
                  102
                  103
                  104
                  105
                  106
                  107
                  108
                  109
                  110
                  111
                  112
                  113
                  114
                  115
                  116
                  117
                  118
                  119
                  120
                  121
                  122
                  123
                  124
                  125
                  126
<snip>

It turns out I have a Mellanox card, too:

# lspci|grep Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

I needed vlan filtering in the bridge, so I worked around it by specifying only the VLAN IDs I need rather than deleting those parameters altogether:

    bridge-vlan-aware yes
    bridge-vids 5 6 7 8 550 551

As an added bonus, the output of bridge vlan show is now nice and short, and it includes the higher VLAN IDs!

# bridge vlan show
<snip>
enp1s0            1 PVID Egress Untagged
                  5
                  6
                  7
                  8
                  550
                  551