VLAN Bridge: What Is The Correct Way?

[Elfy]

Hey all,

Been working on setting up some VLANs as bridges at the host level. Typically I simply add the NIC to the VM, then VLAN tag it there, however I want to try to add the VLAN to the host as a bridge, and pass tagged traffic through a bridge to the VM from there (kind of like VMWare). Firstly, is this even possible? Official documentation seems to indicate it is: https://pve.proxmox.com/wiki/Network_Configuration#sysadmin_network_vlan.

Secondly, I've got two configs, neither of which seem to work. Can someone help me determine if there's something wrong with my approach(es) and which is most correct? Ideally I'd use Config 1 because IMO it is the cleanest. Thanks in advance for any feedback.

Spoiler: Config 1 "traditional VLAN"

auto eno1np0
iface eno1np0 inet manual
#SFP raw NIC

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno1np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#10Gbps Trunk

auto Phones
iface Phones inet manual
        bridge-ports vmbr1.1110
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1110
        vlan-id 1110
#Phones VLAN bridge

auto Security
iface Security inet manual
        bridge-ports vmbr1.1080
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1080
        vlan-id 1080
#Security VLAN bridge

Spoiler: Config 2 "VLAN awareness"

auto eno1np0
iface eno1np0 inet manual
#SFP raw NIC

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno1np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#10Gbps Trunk

auto vlan1110
iface vlan1110 inet manual
        vlan-raw-device vmbr1
#Phones VLAN

auto Phones
iface Phones inet manual
        bridge-ports vlan1110
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1110
#Phones VLAN bridge

auto vlan1080
iface vlan1080 inet manual
        vlan-raw-device vmbr1
#Security VLAN

auto Security
iface Security inet manual
        bridge-ports vlan1080
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1080
#Security VLAN bridge

 

[shanreich]

The first one is the 'better' approach, but you need to remove the following lines from the config of Phones / Security:

        bridge-vlan-aware yes
        bridge-vids 1110
        vlan-id 1110

Since you already are bridging vmbr1.1110 you don't need to configure vlan-awareness / vids on the bridge, since the traffic runs untagged over the bridge and only gets tagged as soon as it leaves the bridge-port.

 

[Elfy]

Great, thanks for the clarification. Now if I use, for example, the Phones bridge on a VM, I would NOT need to tag it, correct? This is my ultimate goal.


Later Edit: when I do bridge vlan show I can tell that Phones is not sending tagged traffic on 1110. So I would have to tag at the VM level, which is redundant. Do you have any idea what might be going wrong? Here's the output of bridge vlan show:

vmbr0             1 PVID Egress Untagged
vmbr1             1 PVID Egress Untagged
                  1060
                  1070
                  1110
vmbr1.1110        1110 PVID Egress Untagged
Phones            1 PVID Egress Untagged
eno1np0.1080      1080 PVID Egress Untagged
Security          1 PVID Egress Untagged

My /etc/network/interfaces for Phones bridge now looks like this:

auto Phones
iface Phones inet manual
        bridge-ports vmbr1.1110
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1110
        bridge-pvid 1110

 

[shanreich]

This should be fine, are you checking via tcpdump on vmbr1? As long as the traffic goes across the Phones bridge it will stay untagged. Only when it leaves the bridge Phones via the bridge port (vmbr1.1110), it will get tagged with 1110. You can check via tcpdumping on vmbr1, if you haven't already.

vmbr1.1110        1110 PVID Egress Untagged

PVID means that untagged frames entering the interface (from the Phones bridge), are getting tagged with VLAN Tag 1110, Egress Untagged means that frames leaving the interface (into the Phones bridge) are getting untagged - the behavior I described above.

 

[Elfy]

I've spent way too many hours on this at this point messing around with dozens of /etc/network/interface configurations. I have concluded that unless I specifically tag the VLAN at the VM's nic (as pictured below), tagged traffic won't flow, despite supposedly the Phones bridge being configured to send untagged to vmbr1.1110, which then should send out the tagged traffic. TCPDumps confirm that packets out of vmbr1.1110 are indeed tagged.

This works:


This does not:


Furthermore, unless I tag it at the VM level, bridge vlan show does not show the VM interface (tap101i1) listed as having a PVID of 1011, which to me is strange behavior...

brctl show shows the interface assigned to the Phones bridge:

root@pve1:~# brctl show
bridge name     bridge id               STP enabled     interfaces
Phones          8000.ac1f6b593800       no              fwpr108p0
                                                        tap101i1
                                                        vmbr1.1110

Is there truly no way to assign a PVID to a bridge so I don't have to explicitly remember my Phones VLAN tag is 1011??

 

[Elfy]

@shanreich Well, you were right all along. Config 1 with your suggestion worked. I have been fighting against a misconfigured DHCP server this whole time and it was handing out IP addresses on different subnets. I've corrected the DHCP server issue, and now the bridges are working. Thank you for your help, I could not have done this without your initial suggestion! My working network config now looks like this:
cat /etc/network/interfaces

auto lo
iface lo inet loopback

auto enp96s0f1np1
iface enp96s0f1np1 inet manual
#SFP to Eth

auto eno1np0
iface eno1np0 inet manual
#SFP raw NIC

auto vmbr0
iface vmbr0 inet static
        address 10.18.10.2/24
        gateway 10.18.10.1
        bridge-ports enp96s0f1np1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#1Gbps Trunk

auto vmbr1
iface vmbr1 inet manual
        bridge-ports eno1np0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
#10Gbps Trunk

auto Phones
iface Phones inet manual
        bridge-ports vmbr1.1110
        bridge-stp off
        bridge-fd 0
#Phones VLAN bridge

auto Security
iface Security inet manual
        bridge-ports vmbr1.1080
        bridge-stp off
        bridge-fd 0
#Security VLAN bridge

auto vlan1070
iface vlan1070 inet static
        address 10.18.7.2/28
        vlan-raw-device vmbr1
#Storage interface

auto vlan1060
iface vlan1060 inet static
        address 10.18.6.2/28
        vlan-raw-device vmbr1
#Corosync interface

source /etc/network/interfaces.d/*

 

[Elfy]

Need to revive this thread again with another hopefully simple question. I'm working on a completely different cluster, trying to create a VLAN bridge called External near the bottom of /etc/network/inferfaces as I did successfully above. Inexplicably neither version of the interfaces config is working on this machine:

Spoiler: Not working with vmbr0.10

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto ens1f0np0
iface ens1f0np0 inet static
    mtu 9000
#OpenFabric1

auto ens1f1np1
iface ens1f1np1 inet static
    mtu 9000
#OpenFabric2

auto vmbr1
iface vmbr1 inet static
    address 10.11.0.4/28
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
#Storage

auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#VLAN Trunk

auto vlan10
iface vlan10 inet static
    address 10.10.0.4/24
    gateway 10.10.0.1
    vlan-raw-device vmbr0
#Servers

auto vlan12
iface vlan12 inet static
    address 10.12.0.4/28
    vlan-raw-device vmbr0
#Cluster

auto vmbr0.10
iface vmbr0.10 inet static
        vlan-raw-device vmbr0

auto External
iface External inet static
    bridge-ports vmbr0.10
    bridge-stp off
    bridge-fd 0
#This bridge also causes networking to fail

post-up /usr/bin/systemctl restart frr.service

This throws an error:
Aug 13 09:56:20 /usr/sbin/ifup[496068]: error: External: bridge port vmbr0.10 does not exist

Spoiler: Not working bridge of vlan10

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto ens1f0np0
iface ens1f0np0 inet static
    mtu 9000
#OpenFabric1

auto ens1f1np1
iface ens1f1np1 inet static
    mtu 9000
#OpenFabric2

auto vmbr1
iface vmbr1 inet static
    address 10.11.0.4/28
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
#Storage

auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#VLAN Trunk

auto vlan10
iface vlan10 inet static
    address 10.10.0.4/24
    gateway 10.10.0.1
    vlan-raw-device vmbr0
#Servers

auto vlan12
iface vlan12 inet static
    address 10.12.0.4/28
    vlan-raw-device vmbr0
#Cluster

auto External
iface External inet static
    bridge-ports vlan10
    bridge-stp off
    bridge-fd 0
#This bridge causes networking to fail

post-up /usr/bin/systemctl restart frr.service

This is also failing, journalctl shows something about the gateway being on vlan10, can I not create a bridge because this is a default route?

 

[vesalius]

So first vmbr0.10 example you have set up vlan10/vmbr0.10 in three different ways. 1 of them is just to give a static address to proxmox of 10.10.0.4, so all good there. The last 2 include iface External inet static , but no static address. Use iface External inet manual instead. I would keep the auto External section and delete the auto vmbr0.10 section as I think it is superfluous, but this also incorrectly has static instead of manual. The same issue of misusing static instead of manual in your vlan10 auto External example.

 

[Elfy]

Thank you for your quick response. I tried your suggestion, however, it's still erroring:
error: vlan10: cmd '/bin/ip route replace default via 10.10.0.1 proto kernel dev vlan10 onlink' failed: returned 1 (Cannot find device "vlan10")
This is the configuration with your suggestions in place:

Spoiler: Also not working with vmbr0.10

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto ens1f0np0
iface ens1f0np0 inet static
    mtu 9000
#OpenFabric1

auto ens1f1np1
iface ens1f1np1 inet static
    mtu 9000
#OpenFabric2

auto vmbr1
iface vmbr1 inet static
    address 10.11.0.4/28
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
#Storage

auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno2
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
#VLAN Trunk

auto External
iface External inet manual
       bridge-ports vmbr0.10
       bridge-stp off
       bridge-fd 0
#External bridge

auto vlan10
iface vlan10 inet static
    address 10.10.0.4/24
    gateway 10.10.0.1
    vlan-raw-device vmbr0
#Servers

auto vlan12
iface vlan12 inet static
    address 10.12.0.4/28
    vlan-raw-device vmbr0
#Cluster

post-up /usr/bin/systemctl restart frr.service

 

[vesalius]

So do you want the #external bridge and servers to both be on vlan 10 Using vmbr0/eno2?

vmbr0.10 = vlan10 using vlan-raw device vmbr0

If that is what you want try:

auto External
iface External inet inet static
       address 10.10.0.4/24
       gateway 10.10.0.1
       bridge-ports vmbr0.10
       bridge-stp off
       bridge-fd 0
#External bridge & Servers

Then delete the auto vlan10 #servers section, but I am not sure that is what you want to do, if not choose a different vlan than 10 for either External or Servers.

 

[Elfy]

auto External
iface External inet inet static
       address 10.10.0.4/24
       gateway 10.10.0.1
       bridge-ports vmbr0.10
       bridge-stp off
       bridge-fd 0
#External bridge & Servers

@vesalius this was the secret sauce I needed. Brilliant, thank you! I still struggle with the networking config, thank you for making it more clear to me.