802.1ad tagged bridge: frames get out as 0x8100

A

Axl_fzetafs

Active Member
Apr 29, 2020
80
2
28
51
ProxMox version: 7.3-3
I created a bond0 device with 4 interfaces

Bash: 
auto enp6s0f0
iface enp6s0f0 inet manual

auto enp6s0f1
iface enp6s0f1 inet manual

auto enp7s0f0
iface enp7s0f0 inet manual

auto enp7s0f1
iface enp7s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp6s0f0 enp6s0f1 enp7s0f0 enp7s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
        vlan-protocol 802.1ad
        vlan-aware yes
        vlan-vids 2-4094

I created a 802.1ad vlan interface with VLAN ID=101

Bash: 
auto i8021ad_0101
iface i8021ad_0101 inet manual
        mtu 9000
        vlan-id 101
        proto 802.1ad
        vlan-raw-device bond0

I then created a bridge with that 802.1ad vlan interface as a slave
Bash: 
auto vmbr0101
iface vmbr0101 inet manual
        bridge-ports i8021ad_0101
        bridge-stp off
        bridge-fd 0
        mtu 9000
        bridge-vlan-protocol 802.1ad

I verified that the protocol for the vmbr0101 and i802ad_0101 is 0x88a8.
Indeed vmbr0101 is the master for i802ad_0101

Bash: 
root@pve:~# cat /sys/class/net/vmbr0101/bridge/vlan_protocol
0x88a8
root@pve:~# cat /sys/class/net/i8021ad_0101/master/bridge/vlan_protocol
0x88a8
root@pve:~# ip link show dev i8021ad_0101
10: i8021ad_0101@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master vmbr0101 state UP mode DEFAULT group default qlen 1000
    link/ether 00:1c:c4:47:63:31 brd ff:ff:ff:ff:ff:ff

In /etc/network/interfaces I also source all the files in the interfaces.d directory

Bash: 
source /etc/network/interfaces.d/*

I then created a Debian machine with just one interface (no VLAN interface), just a plain ethernet interface (ens18 fyi) and launched
Bash: 
dhclient -d ens18
.
If on the ProxMox host I capture on bond0 I do see the DHCP DISCOVER frames, but they are encapsulated in 802.1q tag (0x8100) instead of 0x88a8.

I guessed the 802.1ad syntax for the interface i8021ad_0101 and bond0 (I'm not sure if such statements are necessary)

If I modify /etc/network/interfaces by hand and I relod all the interfaces "ifrelaod -a -d" I don't see any error messages or any options of unsupported statements in that file.

Where am I wrong?
Moreover where can I find the right sintax for the file /etc/network/interfaces with all the options possibles?

I found

https://pve.proxmox.com/pve-docs/chapter-pvesdn.html#pvesdn_setup_example_qinq

but despite the explanation I do not understand where to create the zones and also I don't see any option on the GUI though the documentation says that it should be doable from there.
Do I have to install plugins? Where do I get them? My host ProxMox doesn't have access to the Internet.

TIA

Alex
 
Last edited:
Maybe all boils down to me not having understood how to create zones, it might be easier than it seems. If somebody could help, that would be great :)
 
Ok, I enabled access to the Internet for the PVE and I followed this thread

https://forum.proxmox.com/threads/t...n-pve-bullseye-inrelease-is-not-signed.98927/

because I ran into issues when updating the packages from the repositories.
Then
Code: 
apt update
apt install libpve-network-perl
because
Code: 
ifupdown2
is already installed in version 7 and above.
I got the submenu for zones.

May be there is no documentation for configuring zones manually from the shell because it's recommended to do everything from the GUI
 
Last edited:
Ok, wrapping up everything to get how I'm supposed to work with zones (since I didn't get info on how to do it without).

In the PVE:
1) Create the bond1;
2) create the bridge vmbr0101 and make it VLAN-aware;

in the DataCentre
3) create a QinQ zone named vmbq101 and link to the local bridge vmbr0101 with tag 101.

Use the local bridge for the network interface the VM is connected to

Hopefully, somebody can review these steps and tell me if it's correct or not.

TIA,

Alex
 
Last edited:
Axl_fzetafs said:
Ok, wrapping up everything to get how I'm supposed to work with zones (since I didn't get info on how to do it without).

In the PVE:
1) Create the bond1;
2) create the bridge vmbr0101 and make it VLAN-aware;

in the DataCentre
3) create a QinQ zone named vmbq101 and link to the local bridge vmbr0101 with tag 101.

Use the local bridge for the network interface the VM is connected to

Hopefully, somebody can review these steps and tell me if it's correct or not.

TIA,

Alex
Click to expand...
you don't need any tag on the local vmbr.

when you create a qinq zone && vnet, with a local vlan-aware bridge, proxmox will create something like

# local VLAN_aware_brige----(service vlan tag defined in zone)----->vlanwarebridge zone-(tag define in vnet)----->vnet

It's working too if you don't have a local vlan-aware vmbr too,
in this case it's tag directly the bond0 defined in local vmbr

# bond--->bond.x(svlan defined in zone)----->vlanwarebridge zone-(tag defined in vnet)----->vnet

you can also use openvswitch for local vmbr, in this case it'll create

# local vmbr ovs--->ovsintport(dot1q-tunnel tag defined in zone)------->vlanawarebrige zone-----(tag defined in vnet)--->vnet



So, just create a local vmbr without any tag, it should work out of the box


Code: 
auto bond0
iface bond0 inet manual
        bond-slaves enp6s0f0 enp6s0f1 enp7s0f0 enp7s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        mtu 9000
        bridge-vlan-aware yes
 
Last edited:
spirit said:
you don't need any tag on the local vmbr.

when you create a qinq zone && vnet, with a local vlan-aware bridge, proxmox will create something like

# local VLAN_aware_brige----(service vlan tag defined in zone)----->vlanwarebridge zone-(tag define in vnet)----->vnet

It's working too if you don't have a local vlan-aware vmbr too,
in this case it's tag directly the bond0 defined in local vmbr

# bond--->bond.x(svlan defined in zone)----->vlanwarebridge zone-(tag defined in vnet)----->vnet

you can also use openvswitch for local vmbr, in this case it'll create

# local vmbr ovs--->ovsintport(dot1q-tunnel tag defined in zone)------->vlanawarebrige zone-----(tag defined in vnet)--->vnet



So, just create a local vmbr without any tag, it should work out of the box


Code: 
auto bond0
iface bond0 inet manual
        bond-slaves enp6s0f0 enp6s0f1 enp7s0f0 enp7s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        mtu 9000
        bridge-vlan-aware yes
Click to expand...
Hi @spirit ,

thanks for your reply, Eventually this is my working config

Code: 
auto bond0
iface bond0 inet manual
        bond-slaves enp6s0f0 enp6s0f1 enp7s0f0 enp7s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
        proto 802.1ad

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-vlan-protocol 802.1ad
        mtu 9000

Then for each VM I attach to vmbr1, I specify the tag, it works wonderfully.

My question now is the following:

I want to achieve the same but presenting to VMs a bridge that has alredy a tag associated, this must not be specified under the VM network device's parameters. If the admin specifies the tag in the network device, things must not work; this necause I don't want the VM admin to specify the tag.

Is this possible?
 
Axl_fzetafs said:
Hi @spirit ,

thanks for your reply, Eventually this is my working config

Code: 
auto bond0
iface bond0 inet manual
        bond-slaves enp6s0f0 enp6s0f1 enp7s0f0 enp7s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
        proto 802.1ad

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-vlan-protocol 802.1ad
        mtu 9000

Then for each VM I attach to vmbr1, I specify the tag, it works wonderfully.

My question now is the following:

I want to achieve the same but presenting to VMs a bridge that has alredy a tag associated, this must not be specified under the VM network device's parameters. If the admin specifies the tag in the network device, things must not work; this necause I don't want the VM admin to specify the tag.

Is this possible?
Click to expand...
Yes, it's possible, this is exactly what the sdn module is doing with "vnet" ;)

you can create another bridge, plugged on vmbr1 with tagged interface

Code: 
auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-vlan-protocol 802.1ad
        mtu 9000

auto vnet100
iface vnet100 inet manual
        bridge-ports vmbr1.100
        bridge-stp off
        bridge-fd 0
 
Ah! Thanks!
I thought I would use the primary interface with its subintf bond0.101 but eventually I got weird results!
I will give it a try straight-away!

Alex
 
Last edited:
I think I need a bit more of guidance, @spirit .

Code: 
auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-vlan-protocol 802.1ad
        mtu 9000

auto vnet101
iface vnet101 inet manual
        bridge-ports vmbr1.101
        bridge-stp off
        bridge-fd 0

Once manipulated the /etc/network/interfaces and reloaded with "ifreload -a -d", I don't see the interface vnet101 among the ones available to the VM.

Alex
 
Axl_fzetafs said:
I think I need a bit more of guidance, @spirit .

Code: 
auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        bridge-vlan-protocol 802.1ad
        mtu 9000

auto vnet101
iface vnet101 inet manual
        bridge-ports vmbr1.101
        bridge-stp off
        bridge-fd 0

Once manipulated the /etc/network/interfaces and reloaded with "ifreload -a -d", I don't see the interface vnet101 among the ones available to the VM.

Alex
Click to expand...
The proxmox gui only display "vmbr...".
you can rename vnet101 with vmbr101 if you want. (or use the sdn module, which allow any name ;)
 
spirit said:
The proxmox gui only display "vmbr...".
you can rename vnet101 with vmbr101 if you want. (or use the sdn module, which allow any name ;)
Click to expand...
Ok, for time being I will change the name, I will then investigate the SDN module, which I had a look at a while back.
At that time was there were some grey areas about using the SDN module, I will continue investigating. Thank again!
 
Weird, after having modified the /etc/network/interfaces file the bond0 interface is not recognized any more and in turn all the interfaces linked to it are not recognized, hence the VMS attached to such interfaces won't start.


Code: 
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto enp6s0f0
iface enp6s0f0 inet manual

auto enp6s0f1
iface enp6s0f1 inet manual

auto enp7s0f0
iface enp7s0f0 inet manual

auto enp7s0f1
iface enp7s0f1 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp6s0f0 enp6s0f1 enp7s0f0 enp7s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
        bridge_vids 2-4094
        bridge_vlan_aware 1
        bridge-vlan-protocol 802.1ad
#     proto 802.1ad

auto vmbr0
iface vmbr0 inet static
        address 192.168.41.212/24
        gateway 192.168.41.250
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 1500

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000
        bridge-vlan-protocol 802.1ad

auto vmbr0101
iface vmbr101 inet manual
        bridge-ports vmbr1.101
        bridge-stp off
        bridge-fd 0

Is there aything wrong i nthe way I followed the suggestions?
When trying to fix the issue from the GUI, here below the diff proposed

Code: 
--- /etc/network/interfaces    2023-02-17 14:18:16.286138372 +0100
+++ /etc/network/interfaces.new    2023-02-17 14:19:14.014774153 +0100
@@ -61,8 +61,14 @@
     mtu 9000
     bridge-vlan-protocol 802.1ad
 
-auto vmbr0101
 iface vmbr101 inet manual
     bridge-ports vmbr1.101
     bridge-stp off
     bridge-fd 0
+
+auto vmbr0101
+iface vmbr0101 inet manual
+    bridge-ports none
+    bridge-stp off
+    bridge-fd 0
+
 
Last edited:
I cannot get consistent result in applying the second method. With two VMs, once attached to vmbr1 and the tag 101 specified, and the other one attached to the vmbr101, withthe foirst one DHCP works correctly, on t he seecond it doesn't. the DHCP is located outside the ProxMox server.
With the second one on the bond0 the DHCP frames get out with 802.1q (0x8100) instead of 802.1ad (0x88a8)
 
Axl_fzetafs said:
Is there aything wrong i nthe way I followed the suggestions?
When trying to fix the issue from the GUI, here below the diff proposed

Code: 
--- /etc/network/interfaces    2023-02-17 14:18:16.286138372 +0100
+++ /etc/network/interfaces.new    2023-02-17 14:19:14.014774153 +0100
@@ -61,8 +61,14 @@
     mtu 9000
     bridge-vlan-protocol 802.1ad
 
-auto vmbr0101
 iface vmbr101 inet manual
     bridge-ports vmbr1.101
     bridge-stp off
     bridge-fd 0
+
+auto vmbr0101
+iface vmbr0101 inet manual
+    bridge-ports none
+    bridge-stp off
+    bridge-fd 0
+
Click to expand...
"auto vmbr0101" vs "iface vmbr101"
 
Axl_fzetafs said:
I cannot get consistent result in applying the second method. With two VMs, once attached to vmbr1 and the tag 101 specified, and the other one attached to the vmbr101, withthe foirst one DHCP works correctly, on t he seecond it doesn't. the DHCP is located outside the ProxMox server.
With the second one on the bond0 the DHCP frames get out with 802.1q (0x8100) instead of 802.1ad (0x88a8)
Click to expand...
use the sdn module... (at least, you can use it to generate local configuration, it'll a create /etc/network/interfaces.d/sdn to have working example).
I never managed tagging on bond0 and tagging on vlanware.

Generally, it's better to only tag on bond0.X.Y or link mutiple vlan-aware bridge.

Basically, a vlan tag is added each time the packet is going through a new bridge

vlanaware bridge:

vm-->vnet----(vmbr1.100)--->vmbr1-----(vmbr2.200)-->vmbr2--->bond0-----> outside

or

tagging physical interface:

vm-->vnet-->bond.200.100---->bond0---->outside
 
Hi @spirit ,
thanks for the suggestions. Just a minor error in the docs to fix imo:

Code: 
id: qinqzone1
bridge: vmbr0
service vlan: 20

is wrong because the ID field can be only 8 characters long.
 
Last edited:
Axl_fzetafs said:
Hi @spirit ,
thanks for the suggestions. Just a minor error in the docs to fix imo:

Code: 
id: qinqzone1
bridge: vmbr0
service vlan: 20

is wrong because the ID field can be only 8 characters long.
Click to expand...
thanks for the report. The gui should correctly forbid more than 8 characters.
 
spirit said:
thanks for the report. The gui should correctly forbid more than 8 characters.
Click to expand...
I guess the limit comes from IFNAMSIZ in Linux (max 16 digits for the interfaces' name), isn't? And due to the automatic naming we want to avoid to trespass that limit.
One question though; once added Vnets I see that ProxMox/Linux (I don't know who owns the merit) uses veths and the OVS approach. I see prefixes like sv_, ln_, pr_ but I don't get their meaning. Would you know it? If devs have given such names there might be a reason to distinguish roles or to know the logic behind.

Thx, Alex
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back