VLAN with tag above 126 problem

raku · Aug 8, 2018

I'm using Proxmox cluster with vlan aware bridge configured. Till now, I've used only low number vlan tags (4-5). But right now I need to run a VM in VLAN 551. I can configure VMs network interface, set tag 551. But that's all I can. No traffic is going out from that VLAN. When I change tag to 4 or 5 - interface works OK.

I tried to manually run VLAN 551 on Proxmox host:

Code:

# /etc/network/interfaces
iface ens15d1.551 inet static
   address 172.16.110.70
   netmask 255.255.255.192

Code:

ifup ens15d1.551
RTNETLINK answers: No space left on device
ifup: ignoring unknown interface ens15d1.551=ens15d1.551

After some tests, I realized that max VLAN number I can use is 126:

Code:

ifup ens15d1.126

no errors here

Code:

ifconfig ens15d1.126
ens15d1.126: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.110.70  netmask 255.255.255.192  broadcast 172.16.110.127
        inet6 fe80::268a:7ff:fe75:b1d1  prefixlen 64  scopeid 0x20<link>
        ether 24:8a:07:75:b1:d1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 586 (586.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Has any one of you noticed similar problem?

dietmar · Aug 8, 2018

I am unable to reproduce that (works for me).

raku · Aug 8, 2018

Looks like this is a network card driver issue. Server has got 2 x 1 Gbit Intel network adapter + Mellanox ConnectX-3 Pro 2 x 10 Gbit network adapter.
No issues with running VLAN 551 on Intel cards.

Stoiko Ivanov · Aug 8, 2018

What kind of NIC is that? (haven't seen ensXXdY until now)
could you post the output of `lspci -nnk` (mostly for the kernel module in use)?

The no space left on device might indicate a limitation to 7 bit for vlan-tags in that particular NIC

raku · Aug 8, 2018

Code:

03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
   Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:000c]

More fun with it:
This adapter has got two interfaces: ens15 (I use it as storage network connected to FreNAS with ZFS over iSCSI) ) and ens15d1 (bridge for VMs).
I've just run vlan 551 on ens15. Still no luck with ens15d1

Looks like I'm in need of replacing storage and VM networks.

Stoiko Ivanov · Aug 8, 2018

the kernel-driver would have been helpful (it's included in the output of lspci -nnk).

short searching brought up the following: http://www.mellanox.com/related-docs/prod_software/Mellanox_EN_for_Linux_User_Manual_v4_4.pdf
it seems there might actually be an (old) limitation in the driver. - maybe you could change the behaviour with a module-parameter?

raku · Aug 8, 2018

I've cut too much in previous post

Code:

03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
   Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:000c]
   Kernel driver in use: mlx4_core
   Kernel modules: mlx4_core

Switching interfaces didn't help. I found out that I cannot run VLAN 551 when interface is configured as bridge. When I stopped and destroyed bridge, VLAN 551 could be started.

Stoiko Ivanov · Aug 8, 2018

Sadly I don't have such a NIC lying around, so I cannot reproduce the problem...

any helpful logs in journal/dmesg?

raku · Aug 8, 2018

Logs are empty.

mir · Aug 8, 2018

Have you tried Mellanox's own Linux driver?
http://www.mellanox.com/page/products_dyn?product_family=27

udo · Aug 8, 2018

raku said:
I'm using Proxmox cluster with vlan aware bridge configured. Till now, I've used only low number vlan tags (4-5). But right now I need to run a VM in VLAN 551. I can configure VMs network interface, set tag 551. But that's all I can. No traffic is going out from that VLAN. When I change tag to 4 or 5 - interface works OK.

I tried to manually run VLAN 551 on Proxmox host:

Code:

# /etc/network/interfaces iface ens15d1.551 inet static address 172.16.110.70 netmask 255.255.255.192

Code:

ifup ens15d1.551 RTNETLINK answers: No space left on device ifup: ignoring unknown interface ens15d1.551=ens15d1.551

After some tests, I realized that max VLAN number I can use is 126:

Code:

ifup ens15d1.126

no errors here

Code:

ifconfig ens15d1.126 ens15d1.126: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 172.16.110.70 netmask 255.255.255.192 broadcast 172.16.110.127 inet6 fe80::268a:7ff:fe75:b1d1 prefixlen 64 scopeid 0x20<link> ether 24:8a:07:75:b1:d1 txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 7 bytes 586 (586.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Has any one of you noticed similar problem?

Hi·
that sounds, that you use an cheap switch, which is limited to a small amount of vlans.

I know that hp has such crab switches, which can't use vlans like 1200... they are cheap but also broken!

Udo

mir · Aug 8, 2018

mlx4_core has this option: log_num_vlan
"Log2 max number of VLANs per ETH port (0-7) (int)"
You could try to see if using this changes anything.

Stoiko Ivanov · Aug 9, 2018

Another option would be to upgrade the firmware on the NIC: http://www.mellanox.com/page/firmware_table_ConnectX3ProEN

raku · Aug 9, 2018

@udo - We use Huawei CE6810 for about $4000

It supports 4k VLANs. VLAN for my storage network (FreeNAS -> Proxmox) is on VLAN 4063.

I've found simple workaround of my problem - I receive VLAN 551 on uplink port of the switch and translate it to VLAN 51 which works fine with Proxmox and NIC.

I'll test log_num_vlan option, Mellanox drivers and firmware upgrade at the end of August. Tomorrow I start my 2-weeks vacations

Thanks for your support. I'll let you know when I find the solution of my problem.

spirit · Aug 13, 2018

Hi,
I just trigger this bug with and old server, migrate from proxmox4 to proxmox5, with connectx3.

with bridge vlan aware enable, and bond0 enslaves to vmbr0.

# bridge vlan add dev bond0 vid 2-4094
RTNETLINK answers: No space left on device

I'll try to upgrade firmware

spirit · Aug 13, 2018

upgrade firmware don't help, I'll try with 4.13 kernel (4.15.18 kernel trigger this bug, and it was working fine with 4.4 kernel)

spirit · Aug 13, 2018

4.13 don't work too.

Seem that connectx3 is limited to 128 vlan in hardware.
maybe newer kernel try to offload vlan and not 4.4 kernel, that why it was working before ?

Stoiko Ivanov · Aug 13, 2018

Thanks for reproducing/verifying!

any difference with the module parameters (log_num_vlan )?
does the nic have a vlan-offloading feature you might disable via ethtool (-k/-K)?

spirit · Aug 13, 2018

Stoiko Ivanov said:
Thanks for reproducing/verifying!

any difference with the module parameters (log_num_vlan )?
does the nic have a vlan-offloading feature you might disable via ethtool (-k/-K)?

log_num_vlan -> is deprecated, and max value is 7 (2^7->128), so it don't help

I'm currently trying with ethtool, I'm seeing some differences. (I have also opened a case to mellanox support).

I'll reply in this thread when I'll more infos.

mir · Aug 13, 2018

spirit said:
log_num_vlan -> is deprecated, and max value is 7 (2^7->128), so it don't help

This might indicate that only 128 vlans is supported in hardware but I would expect this to be a relative number and not limited to 1-129

VLAN with tag above 126 problem

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Famous Member

Distinguished Member

Famous Member

Proxmox Staff Member

Renowned Member

Distinguished Member

Distinguished Member

Distinguished Member

Proxmox Staff Member

Distinguished Member

Famous Member