Proxmox VE and VLAN tag stacking

skraw

New Member
Aug 13, 2019
10
0
1
52
Hello all,

can anyone confirm that VLAN tag stacking works with Proxmox VE and bridging configured as "vlan-aware"?
We are trying to make the following work:
Many VLANs arrive at the host encapsulated in an outer VLAN 1008.
Host bridge is vlan-aware. Guest gets an interface with tag=1008. So the outer VLAN tag should be stripped away. Guest handles all inner VLANs with separate vlan-interfaces. Outgoing traffic (of all guest vlan interfaces) is again encapsulated in VLAN1008 and switched to its destination (outside host).
Does this work as expected?
--
Regards,
Stephan

PS: Let me explain this more detailed. It looks to us like the way in to the guest does in fact work. The (outer) tag is stripped and the vlans seem to be visible to the guest. But on the way out from the guest the outer tag does not seem to be added again. But this is exactly the feature we need. A tag has to be added back on outgoing packets no matter if they are already tagged or not. This is kind of a question around the tap-device. We found no valid information on the net about tag stacking with tap devices ...
 
Last edited:

spirit

Well-Known Member
Apr 2, 2010
3,381
140
63
www.odiso.com
with vlan-aware brige, it's a little bit complex, because you need 2 bridge for each vlan.

I think it's working with ifupdown2

Code:
auto vmbr0
iface vmbr0 inet manual
       bridge-ports eth0
       bridge-stp off
       bridge-fd 0
       bridge-vlan-aware yes

auto vmbr1
iface vmbr1 inet manual
       bridge-ports vmbr0.408
       bridge-stp off
       bridge-fd 0
       bridge-vlan-aware yes

But it can work with non vlan aware bridge too

Code:
auto vmbr0
iface vmbr0 inet manual
       bridge-ports eth0.1008
       bridge-stp off
       bridge-fd 0
then, when you add vlan "X" to a vm, proxmox will create a vmbr0vX with eth0.1008.X
 
  • Like
Reactions: Stoiko Ivanov

skraw

New Member
Aug 13, 2019
10
0
1
52
Ok, I can confirm that your second code part works. I tried that.
Unfortunately it does not solve the problem. It is the details that are complex:
- I need one vlan-aware bridge for guests that should be attached to a single "inner" vlan, basically needing an interface like eth0.1008.25.
- On the other hand I need a bridge for the outer vlan to attach guests that should handle the inner vlans themselves, meaning an interface like eth0.1008 - and I mean _for the same outer vlan_.
Since you cannot create two bridges with the same bridge-port it is vital that one bridge can handle both, the vlan-awareness and the simple bridging of all vlans to one (another) guest.
Is it possible to attach a guest to a vlan-aware bridge without tagging a certain vlan for it (so it simply gets all vlans)?

PS: what do you mean by "ifupdown2"?
 

spirit

Well-Known Member
Apr 2, 2010
3,381
140
63
www.odiso.com
I think this should work

withtout vlan aware:

Code:
auto vmbr0
iface vmbr0 inet manual
       bridge-ports eth0.1008
       bridge-stp off
       bridge-fd 0

auto vmbr0v25 
iface vmbr0v25 inet manual
       bridge-ports eth0.1008.25
       bridge-stp off
       bridge-fd 0


PS: what do you mean by "ifupdown2"?
ifupdown2 is a new package, which replace ifupdown. this is the software which manage /etc/network/interfaces to create/manage interfaces.
an ifupdown2 support more syntax. (apt install ifupdown2)
 

skraw

New Member
Aug 13, 2019
10
0
1
52
Hm, which means I would have to hand-configure all inner vlans needed by guests as bridges on every host of the cluster...
I guess that's what "vlan-aware" was intended to replace :)

Just checked: I have installed:
ifupdown/stable,now 0.8.35 amd64 [installed]
 

spirit

Well-Known Member
Apr 2, 2010
3,381
140
63
www.odiso.com
Hm, which means I would have to hand-configure all inner vlans needed by guests as bridges on every host of the cluster...
I guess that's what "vlan-aware" was intended to replace :)
How/where do your guest configure the inner vlan ? on the proxmox gui ? or in the vm, tagging interface directly ?

For the proxmox gui, my solution should work,
users can use vmbr0 for the vm interface, and choose inner vlan tag.
user can choose vmbr0v25, where the inner tag is already defined.
 

skraw

New Member
Aug 13, 2019
10
0
1
52
There are two use-cases by the guests:
1) One type of guest uses the bridged outer vlan and tags the inner vlan itself inside the guest linux
2) Another type of guest uses inner vlan host bridges as simple interfaces to communicate to type 1 guests vlan interface (defined inside the guest)

The host bridges from 2) are your vmbr0v25 example equivalent.
The bridged outer vlan from 1) is your vmbr0

If there are 100 guests of type 2, and you have 10 hosts, there are 1000 additional bridges to define in /etc/network/interfaces (100 per host). Working but not very nice to look at.
You think that will work with standard ifupdown package?
 

skraw

New Member
Aug 13, 2019
10
0
1
52
Ok, the double vlan tagged interface definition does not work:

# ifup enp5s0f0.1007.8
Error: 8021q: VLAN device already exists.
ifup: ignoring unknown interface enp5s0f0.1007.8=enp5s0f0.1007.8
 

skraw

New Member
Aug 13, 2019
10
0
1
52
PS: I tried to install your ifupdown2 package but shot down a cluster node by doing that. I do not quite understand why it takes down the whole node and restarts just because of this installation ...

PS2: After rebooting the node the dual tagged interface came up, which probably means ifupdown2 works.
Interestingly your bridge definition as "vmbr0v25" (or the like) shows "unknown" in the PMX GUI as Interface type. So I renamed it to "vmbr5" and then it shows "linux bridge".
 
Last edited:

skraw

New Member
Aug 13, 2019
10
0
1
52
Now we encountered other problems:

1) ARP between the external inner vlan and the newly created bridge does not work, in fact it seems indeed no arp at all works. The guest with its own vlan config (from other bridge) cannot arp to the guest on the inner vlan host bridge and both cannot arp to an external switch.

2) the interface naming is a problem. Interface names are required to be 15 chars or shorter. We tried to escape problem 1 by configuring a new interface enp5s0f0.1007.28 (instead of 8) and got an error for name length on ifup.
 

spirit

Well-Known Member
Apr 2, 2010
3,381
140
63
www.odiso.com
PS: I tried to install your ifupdown2 package but shot down a cluster node by doing that. I do not quite understand why it takes down the whole node and restarts just because of this installation ...
it need to to restart network, but whole node I'm not sure ..


Interestingly your bridge definition as "vmbr0v25" (or the like) shows "unknown" in the PMX GUI as Interface type. So I renamed it to "vmbr5" and then it shows "linux bridge".
be carefull that if you define a vlan25 in a vm on vmbr0, proxmox will create vmbr0v25, and it could break other vmbr5.

1) ARP between the external inner vlan and the newly created bridge does not work, in fact it seems indeed no arp at all works. The guest with its own vlan config (from other bridge) cannot arp to the guest on the inner vlan host bridge and both cannot arp to an external switch.
mmm, that seem the correct behaviour ? arp and other network traffic can only work in the same inner+outer vlan .

2) the interface naming is a problem. Interface names are required to be 15 chars or shorter. We tried to escape problem 1 by configuring a new interface enp5s0f0.1007.28 (instead of 8) and got an error for name length on ifup.
maybe try to use old ethX naming, adding "net.ifnames=0 biosdevname=0" to grub option ?




BTW, I'm working on a new proxmox network management (sdn feature), at datacenter level. (define once, it's apply to all nodes).
It's not yet ready, but it should help a lot for your case. (it's suporting vxlan, vlan stack, ifupdown2 + dynamic reload,....)
I think it should be available soon (2-3months)
 

skraw

New Member
Aug 13, 2019
10
0
1
52
it need to to restart network, but whole node I'm not sure ..



be carefull that if you define a vlan25 in a vm on vmbr0, proxmox will create vmbr0v25, and it could break other vmbr5.


mmm, that seem the correct behaviour ? arp and other network traffic can only work in the same inner+outer vlan .
No it is not. The guest attached to the outer vlan does of course create itself several vlans (the inner ones). And inside these vlans only those arp work that are not used in the host bridge attached to the inner vlan. Believe me that I know a thing or two about networking, and I have already debugged an equivalent setup based on qemu and bridges years ago.
Arp is really completely broken in this use case. The Arp requests are not even reaching the guest with its self created inner vlan interface. I thought at first it works, but the arp tables were left over from the old setup with singular qemus and bridges on another host inside the same vlan. Since we did not change the macs while transfering the guest to proxmox it did work. But today the ifupdown2 packages made all cluster nodes reboot during installation and after that arp was dead for this case. As soon as we detached the host bridge based on the double tagged interface everything worked again with arp inside this very vlan.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!