VLAN with tag above 126 problem

response from mellanox support:

"
Based on the information provided, our engineers came back with the following explanation:

"With the older kernels, the driver was never able to report the error regarding the limitation, this was resolved in later kernels. Still, even being able to configure > 128 vlan's, the limitation for ConnectX-3 is still 128 vlan's"

Please update the case if you need any further assistance regarding this request or we can proceed with the closing of the case.
"

:/
 
Not only restricted in number of vlans but also restricted in the vlan numbering. If they have to choose a byte to store vlans why choose a signed byte! There is no such thing as a negative vlan number.
 
Ok - learned something new - w.r.t. limitation of drivers ... :/

just out of curiosity: does disabling the offloading features change anything (if someone has time/is interested)/make vlans above 126 configurable?
(ethtool -k $nic - to list the available ones)
 
Bumping

@Stoiko Ivanov - any specific offloading feature?

I've disabled anything I was able:
Code:
ethtool -k ens15d1 |grep offload
Cannot get device udp-fragmentation-offload settings: Operation not supported
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off

Still not able to configure vlan over 126.

Looks like I have to get used to vlan mapping on the switch :-/
 
I thought about the rx-vlan/tx-vlan -offload settings primarily (helped me with a different NIC once...).
If you still want to experiment - you could take a look at the other features (i.e. without the |grep offload) - AFAIR the offered flags vary from driver-version (thus kernel version) to version, and also change with newer versions of ethtool.
 
I've disabled anything with vlan in name:
Code:
ethtool -k ens15d1 |grep vlan
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-vlan-offload: off
tx-vlan-offload: off
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off [requested on]
rx-vlan-stag-filter: on [fixed]

No luck.

Here's everything what ethtool returns:
Code:
ethtool -k ens15d1
Features for ens15d1:
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-checksumming: on
tx-checksumming: on
   tx-checksum-ipv4: on
   tx-checksum-ip-generic: off [fixed]
   tx-checksum-ipv6: on
   tx-checksum-fcoe-crc: off [fixed]
   tx-checksum-sctp: off [fixed]
scatter-gather: on
   tx-scatter-gather: on
   tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
   tx-tcp-segmentation: on
   tx-tcp-ecn-segmentation: off [fixed]
   tx-tcp-mangleid-segmentation: off
   tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off [requested on]
rx-vlan-stag-filter: on [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: on
 
Hi,

I have ProxMox (pve-manager/4.4-13/7ea56165 (running kernel: 4.4.44-1-pve)) with Mellanox ConnectX-3Pro NIC (MCX312B-XCCT). The firmware is 2.35.5100 , and in kernel driver driver: mlx4_en version: 2.2-1 (Feb 2014). I don't have any issue with VLANs.

I am installing the ProxMox 5.2 on other server with 2.42.5000 and there it is not working.
 
So I changed the configuration of the bridge:
auto vmbr0
iface vmbr0 inet manual
bridge-ports enp3s0d1
bridge-stp off
bridge-fd 0

And restart the server. Now it is working with ProxMox 5 and 2.42.5000 firmware and in kernel 2.2 driver.
 
@mkunin: what exactly did you change? Do you mean the perdictable network interface names (enp3s0d1 instead of eth0)?
 
I think he removed:
Code:
   bridge-vlan-aware yes
   bridge-vids 2-4094
 
I think he removed:
Code:
   bridge-vlan-aware yes
   bridge-vids 2-4094

the problem with last mellanox driver, with bridge vlan aware, is that vlans are allocated from 2 to 4096 directly,
and it's blocking around 126-128.

without bridge vlan aware, you can allocated the vlan you want (but still limited to 128 total vlan numbers)
 
I got to this thread googling "proxmox vlan 126" after determining that a new vlan I was adding (also in the 500 range, coincidentally) was not passing traffic between the nic and proxmox bridge, and stopped at 126:

Code:
# bridge vlan show vmbr0
port              vlan-id 
<snip>
enp1s0            1 PVID Egress Untagged
                  2
                  3
                  4
                  5
                  6
                  7
                  8
                  9
                  10
                  11
                  12
                  13
                  14
                  15
                  16
                  17
                  18
                  19
                  20
                  21
                  22
                  23
                  24
                  25
                  26
                  27
                  28
                  29
                  30
                  31
                  32
                  33
                  34
                  35
                  36
                  37
                  38
                  39
                  40
                  41
                  42
                  43
                  44
                  45
                  46
                  47
                  48
                  49
                  50
                  51
                  52
                  53
                  54
                  55
                  56
                  57
                  58
                  59
                  60
                  61
                  62
                  63
                  64
                  65
                  66
                  67
                  68
                  69
                  70
                  71
                  72
                  73
                  74
                  75
                  76
                  77
                  78
                  79
                  80
                  81
                  82
                  83
                  84
                  85
                  86
                  87
                  88
                  89
                  90
                  91
                  92
                  93
                  94
                  95
                  96
                  97
                  98
                  99
                  100
                  101
                  102
                  103
                  104
                  105
                  106
                  107
                  108
                  109
                  110
                  111
                  112
                  113
                  114
                  115
                  116
                  117
                  118
                  119
                  120
                  121
                  122
                  123
                  124
                  125
                  126
<snip>

It turns out I have a Mellanox card, too:

Code:
# lspci|grep Mellanox
01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

I needed vlan filtering in the bridge, so I worked around it by specifying only the VLAN IDs I need rather than deleting those parameters altogether:

Code:
    bridge-vlan-aware yes
    bridge-vids 5 6 7 8 550 551

As an added bonus, the output of bridge vlan show is now nice and short, and it includes the higher VLAN IDs!

Code:
# bridge vlan show
<snip>
enp1s0            1 PVID Egress Untagged
                  5
                  6
                  7
                  8
                  550
                  551
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!