VLAN with tag above 126 problem

raku

Well-Known Member
Apr 16, 2016
42
4
48
43
I'm using Proxmox cluster with vlan aware bridge configured. Till now, I've used only low number vlan tags (4-5). But right now I need to run a VM in VLAN 551. I can configure VMs network interface, set tag 551. But that's all I can. No traffic is going out from that VLAN. When I change tag to 4 or 5 - interface works OK.

I tried to manually run VLAN 551 on Proxmox host:
Code:
# /etc/network/interfaces
iface ens15d1.551 inet static
   address 172.16.110.70
   netmask 255.255.255.192
Code:
ifup ens15d1.551
RTNETLINK answers: No space left on device
ifup: ignoring unknown interface ens15d1.551=ens15d1.551

After some tests, I realized that max VLAN number I can use is 126:
Code:
ifup ens15d1.126
no errors here
Code:
ifconfig ens15d1.126
ens15d1.126: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.110.70  netmask 255.255.255.192  broadcast 172.16.110.127
        inet6 fe80::268a:7ff:fe75:b1d1  prefixlen 64  scopeid 0x20<link>
        ether 24:8a:07:75:b1:d1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 586 (586.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Has any one of you noticed similar problem?
 
Looks like this is a network card driver issue. Server has got 2 x 1 Gbit Intel network adapter + Mellanox ConnectX-3 Pro 2 x 10 Gbit network adapter.
No issues with running VLAN 551 on Intel cards.
 
What kind of NIC is that? (haven't seen ensXXdY until now)
could you post the output of `lspci -nnk` (mostly for the kernel module in use)?

The no space left on device might indicate a limitation to 7 bit for vlan-tags in that particular NIC
 
Code:
03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
   Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:000c]

More fun with it:
This adapter has got two interfaces: ens15 (I use it as storage network connected to FreNAS with ZFS over iSCSI) ) and ens15d1 (bridge for VMs).
I've just run vlan 551 on ens15. Still no luck with ens15d1

Looks like I'm in need of replacing storage and VM networks.
 
I've cut too much in previous post
Code:
03:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
   Subsystem: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:000c]
   Kernel driver in use: mlx4_core
   Kernel modules: mlx4_core

Switching interfaces didn't help. I found out that I cannot run VLAN 551 when interface is configured as bridge. When I stopped and destroyed bridge, VLAN 551 could be started.
 
Sadly I don't have such a NIC lying around, so I cannot reproduce the problem...

any helpful logs in journal/dmesg?
 
I'm using Proxmox cluster with vlan aware bridge configured. Till now, I've used only low number vlan tags (4-5). But right now I need to run a VM in VLAN 551. I can configure VMs network interface, set tag 551. But that's all I can. No traffic is going out from that VLAN. When I change tag to 4 or 5 - interface works OK.

I tried to manually run VLAN 551 on Proxmox host:
Code:
# /etc/network/interfaces
iface ens15d1.551 inet static
   address 172.16.110.70
   netmask 255.255.255.192
Code:
ifup ens15d1.551
RTNETLINK answers: No space left on device
ifup: ignoring unknown interface ens15d1.551=ens15d1.551

After some tests, I realized that max VLAN number I can use is 126:
Code:
ifup ens15d1.126
no errors here
Code:
ifconfig ens15d1.126
ens15d1.126: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.110.70  netmask 255.255.255.192  broadcast 172.16.110.127
        inet6 fe80::268a:7ff:fe75:b1d1  prefixlen 64  scopeid 0x20<link>
        ether 24:8a:07:75:b1:d1  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 586 (586.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Has any one of you noticed similar problem?
Hi·
that sounds, that you use an cheap switch, which is limited to a small amount of vlans.

I know that hp has such crab switches, which can't use vlans like 1200... they are cheap but also broken!

Udo
 
mlx4_core has this option: log_num_vlan
"Log2 max number of VLANs per ETH port (0-7) (int)"
You could try to see if using this changes anything.
 
@udo - We use Huawei CE6810 for about $4000 :) It supports 4k VLANs. VLAN for my storage network (FreeNAS -> Proxmox) is on VLAN 4063.

I've found simple workaround of my problem - I receive VLAN 551 on uplink port of the switch and translate it to VLAN 51 which works fine with Proxmox and NIC.

I'll test log_num_vlan option, Mellanox drivers and firmware upgrade at the end of August. Tomorrow I start my 2-weeks vacations :)

Thanks for your support. I'll let you know when I find the solution of my problem.
 
Hi,
I just trigger this bug with and old server, migrate from proxmox4 to proxmox5, with connectx3.

with bridge vlan aware enable, and bond0 enslaves to vmbr0.

# bridge vlan add dev bond0 vid 2-4094
RTNETLINK answers: No space left on device

I'll try to upgrade firmware
 
Thanks for reproducing/verifying!

any difference with the module parameters (log_num_vlan )?
does the nic have a vlan-offloading feature you might disable via ethtool (-k/-K)?
 
Thanks for reproducing/verifying!

any difference with the module parameters (log_num_vlan )?
does the nic have a vlan-offloading feature you might disable via ethtool (-k/-K)?

log_num_vlan -> is deprecated, and max value is 7 (2^7->128), so it don't help

I'm currently trying with ethtool, I'm seeing some differences. (I have also opened a case to mellanox support).

I'll reply in this thread when I'll more infos.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!