[Solved] SDN: zone set to 802.1Q despite the configuration is 802.1AD? (Proxmox 7.4.3)

A

Axl_fzetafs

Active Member
Apr 29, 2020
80
2
28
51
This is an install from scratch on two nodes. I can reprodice the issue, multiple times.

Code: 
root@pve02:~# cat /etc/pve/sdn/zones.cfg
qinq: q4064
        bridge vmbr0
        tag 4064
        ipam pve
        vlan-protocol 802.1ad

root@pve02:~# cat /etc/pve/sdn/vnets.cfg
vnet: v4064
        zone q4064

root@pve02:~# brctl show
bridge name     bridge id               STP enabled     interfaces
v4064           8000.f643b0682394       no              pr_q4064
vmbr0           8000.901b0efa7138       no              enp1s0
                                                        tap100i0
z_q4064         8000.901b0efa7138       no              ln_q4064
                                                        vmbr0.4064

root@pve02:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff
14: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 3e:e5:68:fe:26:98 brd ff:ff:ff:ff:ff:ff
15: ln_q4064@pr_q4064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master z_q4064 state UP mode DEFAULT group default qlen 1000
    link/ether ea:18:a6:36:45:d1 brd ff:ff:ff:ff:ff:ff
16: pr_q4064@ln_q4064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master v4064 state UP mode DEFAULT group default qlen 1000
    link/ether f6:43:b0:68:23:94 brd ff:ff:ff:ff:ff:ff
17: v4064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether f6:43:b0:68:23:94 brd ff:ff:ff:ff:ff:ff
18: vmbr0.4064@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master z_q4064 state UP mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff
19: z_q4064: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff

If I query the interface vmbr0.4064 I see the protocol is set to 802.1Q rather than 802.1AD (see 3rd line "vlan protocol 802.1Q id 4064 <REORDER_HDR>")

Code: 
root@pve02:~# ip -details link show dev vmbr0.4064
18: vmbr0.4064@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master z_q4064 state UP mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 0 maxmtu 65535
    vlan protocol 802.1Q id 4064 <REORDER_HDR>
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.90:1b:e:fa:71:38 designated_root 8000.90:1b:e:fa:71:38 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 64000 gso_max_segs 64

If I create manually the interface

Code: 
ip link add link vmbr0 name vmbr0.4065 type vlan protocol 802.1ad id 4065

it gets the correct protocol

Code: 
root@pve02:~# ip -details link show dev vmbr0.4065
21: vmbr0.4065@vmbr0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65535
    vlan protocol 802.1ad id 4065 <REORDER_HDR> addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 64000 gso_max_segs 64

Any advice?

Thanks,
Alex
 
Last edited:
QinQ == 802.1ad == 802.1Q (it was not part of the original 802.1Q, but that was updated)

edit: ah, your question is specifically about the last output.. is there something that is not working as expected? if so, please provide details about that.
 
Last edited:
Thanks @fabian.

I will try to stick to the point.
This are my zones/vnets/interfaces/sdn config files

Code: 
root@pve02:~# cat /etc/pve/sdn/zones.cfg
qinq: z4064
        bridge vmbr0
        tag 4064
        ipam pve
        vlan-protocol 802.1ad

root@pve02:~# cat /etc/pve/sdn/vnets.cfg
vnet: v90
        zone z4064
        alias Vlan 90
        tag 90
        vlanaware 1

root@pve02:~# cat /etc/network/interfaces.d/sdn
#version:29

auto ln_z4064
iface ln_z4064
        link-type veth
        veth-peer-name pr_z4064

auto pr_z4064
iface pr_z4064
        link-type veth
        veth-peer-name ln_z4064

auto v90
iface v90
        bridge_ports z_z4064.90
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        alias Vlan 90

auto vmbr0
iface vmbr0
        bridge-vlan-protocol 802.1ad

auto z_z4064
iface z_z4064
        bridge-stp off
        bridge-ports vmbr0.4064 ln_z4064
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

root@pve02:~# cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet static
        address *.*.*.*/23
        gateway *.*.*.*
        bridge-ports enp1s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

source /etc/network/interfaces.d/*

I assigned the interface of the VM to the bridge vnet and set the tag to 227. Then I configured the interface with IP 1.1.1.1/24 and I started a ping to 1.1.1.2. Correctly the hosts sent out ARP requests to know the MAC address of 1.1.1.2.
I started sniffing on vmbr0 (the bridge whose only leg is enp1s0).

vmbr0 <-> z4064 <-> v90 (w/ tag 227) <-> intf of the VM

What I got (tcpdump -nei vmbr0) is the following (I know it's a bit weird to get a packet with three tags, but it's for understanding purposes)

Code: 
 17:34:34.581052 aa:db:5a:20:bf:48 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 54: vlan 4064, p 0, ethertype 802.1Q (0x8100), vlan 90, p 0, ethertype 802.1Q (0x8100), vlan 227, p 0, ethertype ARP (0x0806), Request who-has 1.1.1.2 tell 1.1.1.1, length 28

In other words, moving from the beginning of the frame to its end:
(dst mac, src mac)
ethertype 802.1Q (0x8100), vlan 4064
ethertype 802.1Q (0x8100), vlan 90
ethertype 802.1Q (0x8100), vlan 227

What is not expected is the type of the first tag because I instructed SDN differently, see the snippet from the zones.cfg

Code: 
qinq: z4064
        bridge vmbr0
        tag 4064
        ipam pve
        vlan-protocol 802.1ad

Instead Linux tells me something different (vlan protocol 802.1Q)

Code: 
root@pve02:~# ip -d link show dev vmbr0.4064
31: vmbr0.4064@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master z_z4064 state UP mode DEFAULT group default qlen 1000
    link/ether 90:1b:0e:fa:71:38 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 0 maxmtu 65535
    vlan protocol 802.1Q id 4064 <REORDER_HDR>
    bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.90:1b:e:fa:71:38 designated_root 8000.90:1b:e:fa:71:38 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neig

You see three ethertype 802.1Q (0x8100), stacked one upon the other (in order 4064, 90, 227) whereas, in my opinion it should be ethertype 802.1AD (0x88a8), ethertype 802.1Q (0x8100), ethertype 802.1Q (0x8100), sill with the same order 4064, 90, 227.

In other words I would expect, moving from the beginning of the frame to its end:
(dst mac, src mac)
ethertype 802.1AD (0x88a8), vlan 4064
ethertype 802.1Q (0x8100), vlan 90
ethertype 802.1Q (0x8100), vlan 227

I hope my explanation is clear for you, I have attached the PCAP for more clarity.

Again, three tags won't be seen anywhere in the networld, but the point is right the first tag.

Alex
 

Attachments

Last edited:
[UPDATE]

By issueing the following commands I managed to change it the behaviour and now I get the packets the way I want
Code: 
ip link del vmbr0.4064
ip link add link vmbr0 name vmbr0.4064 type vlan protocol 802.1ad id 4064

ProxMox complains a little bit if I issue "ifreload -a -d", but it's a bed test, so I don't worry that much.

Code: 
error: vmbr0.4064: cannot change vlan-protocol to 802.1q: operation not supported. Please delete the device with 'ifdown vmbr0.4064' and recreate it to apply the change.

I could not manage to use the suggestion reported in the error above.
Of course what I get should be configurable from the GUI, so maybe I'm using it in the wrong way...

This time, the capture gives me what I'm looking for
Code: 
18:58:28.905371 aa:db:5a:20:bf:48 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q-QinQ (0x88a8), length 54: vlan 4064, p 0, ethertype 802.1Q (0x8100), vlan 90, p 0, ethertype 802.1Q (0x8100), vlan 227, p 0, ethertype ARP (0x0806), Request who-has 1.1.1.2 tell 1.1.1.1, length 28

Now I get

(dst mac, src mac)
ethertype 802.1AD (0x88a8), vlan 4064
ethertype 802.1Q (0x8100), vlan 90
ethertype 802.1Q (0x8100), vlan 227
 
Last edited:
I'll look at it tomorrow, sound like a bug.
(I'm pretty sure than vmbr0.Y inherit the vlan-protocol previously, maybe something have change in the kernel)

Should be trivial to fix, I'll send a patch for testing.

Can you open a bugzilla.proxmox.com ?



(BTW, do you really need triple vlan tag ? Do you have tried without vlan-aware vnet && without tag defined on the vm nic ? )
 
Last edited:
  • Like
Reactions: fabian
can you try this package with the fix:

Code: 
wget https://mutulin1.odiso.net/libpve-network-perl_0.7.3_all.deb
dpkg -i libpve-network-perl_0.7.3_all.deb
systemctl restart pveproxy
systemctl restart pvedaemon

it should generate in /etc/network/interfaces.d/sdn , and a new config

Code: 
auto vmbr0.4064
iface vmbr0.4064
        vlan-protocol 802.1ad


Note that indeed switching protocol of this vlan interface with ifreload is not possible currently (kernel limitation) without doing manually an ifdown of the vmbr0.4064 interface.
I'll try to automate the ifdown later.


Please test and report if it's working for you.
(and open the bugzilla)
 
  • Like
Reactions: fabian
Thanks! I will do later today (testing an opening the bug in the tool) as at the moment I cannot.
 
  • Like
Reactions: Axl_fzetafs
i finally been able to get SDN working ok a 7.4.3 cluster by adding on the defined bridge:

in /etc/network/interfaces

bridge-vlan-protocol 802.1ad


if you have a bonding , its important also to specify :

vlan-protocol 802.1ad

for a reason i cannot explain this cluster as never been able to work with 802.1q even if the config is identical as another ending in the same HPE network stack switch.

everthing is identical but cluster is currently 7.3.4 ( the one who work with 802.1q )
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom