[SOLVED] Problem with VLANs over VMBRs

G

gawron737

Member
Oct 3, 2022
17
2
8
Hi

I have a problem in networking in my Proxmox v8. I use multiple VMBRs for each vlan because I want to easy selection vlan for users. When I bind vmbr450 for VM and configure network all works fine but when I want to bind vmbr0 to another VM and configure VM's network in vlan450 using vmbr0 communication doesn't work. I need use vmbr0 in particular cases and configure inside VM vlans but in this configuration PVE networking don't work fine when I use vmbr0. I want to set this up in a way that the virtual machine can receive tagged frames using vmbr0 (where the VLAN is configured internally) – something like a trunk connection. However, other virtual machines should also be able to use VLANs directly, with incoming frames being untagged by that VLAN. The end users use interfaces vmbr (i.e. vmbr450) which are connected with specified vlans. Disable or enable vlan aware it's change nothing.

Anyone could help me in my problem? Below slice of /etc/network/interfaces:

Code:
auto eth6
iface eth6 inet manual
mtu 9000

auto eth7
iface eth7 inet manual
mtu 9000

auto bond0
iface bond0 inet manual
bond-slaves eth6 eth7
bond-miimon 100
bond-mode active-backup
bond-primary eth6
mtu 9000

auto vmbr0
iface vmbr0 inet static
address X.X.X.X/X
gateway X.X.X.X
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000

auto vmbr450
iface vmbr450 inet manual
bridge-ports vlan450
bridge-stp off
bridge-fd 0
mtu 9000

auto vlan450
iface vlan450 inet manual
mtu 9000
vlan-raw-device bond0

In summary, I aim to achieve a state where the interfaces consist of vmbr0, directly connected to bond0, handling all traffic, and VLAN interfaces, also associated with bond0, receiving frames with specific tags. Based on the VLAN interfaces, VMBR interfaces are created, for example, for VLAN 450, there is vmbr450.

On vm with vlan interfaces inside (when configure networking inside VM's communication doesn't work - vlan450): :
1712222272626.png


On another vm, when vmbr450 is connected directly (when configure networking inside VM's communication works fine):
1712222299088.png



Regards
Tom
 
  • Like
Reactions: piotrzu
Hi @gawron737 ,

I'm not sure if the two physical interfaces are assigned different VLANs at the switch and that's why you're bonding them, or if they're exactly the same, but I believe I'm setting up something similar to what you're doing right now. I have a single physical link from my Proxmox host to a switch, which is then connected to my router host. On the router (opnsense) There are several networks setup, and I have at least two networks VLAN-trunked from router to Proxmox. The NIC in Proxmox handles 802.1Q trunking, so I can send multiple tagged vlans to one physical interface. I'm not sure how that works with bonded interfaces, but in practice at least, am I understanding correctly?

Here's what my node network config looks like:

1712522645123.png

The switch port is configured to deliver the two VLANs (30 and 150) to the physical enp4s0 interface, which I then breakout into two Linux VLANs associated with the physical device by name in dot notation only. Then, I create two Bridges. The first one is what I'm using for Proxmox's management interface, so I have it statically assigned an address on my management VLAN. The next bridge is just a network I'm using for one of my VMs to try and make connecting from my machine a little easier. So far everything seems to be working - the Proxmox interface is working fine, and the VM is able to get an IP from my router's DHCP server. I haven't configured the firewall on my router yet so that still has to be done.

I hope this helps and comes close to answering your question.
 
Last edited:
I have bond interface because it's a production environment. Your post don't resolve my problem because I have specyfic case when I use vmbr0 and inside VM's create VLAN, but when I use vmbr450 for another VM all works fine.
 
gawron737 said:
Hi

I have a problem in networking in my Proxmox v8. I use multiple VMBRs for each vlan because I want to easy selection vlan for users. When I bind vmbr450 for VM and configure network all works fine but when I want to bind vmbr0 to another VM and configure VM's network in vlan450 using vmbr0 communication doesn't work. I need use vmbr0 in particular cases and configure inside VM vlans but in this configuration PVE networking don't work fine when I use vmbr0. I want to set this up in a way that the virtual machine can receive tagged frames using vmbr0 (where the VLAN is configured internally) – something like a trunk connection. However, other virtual machines should also be able to use VLANs directly, with incoming frames being untagged by that VLAN. The end users use interfaces vmbr (i.e. vmbr450) which are connected with specified vlans. Disable or enable vlan aware it's change nothing.

Anyone could help me in my problem? Below slice of /etc/network/interfaces:

Code:
auto eth6
iface eth6 inet manual
mtu 9000

auto eth7
iface eth7 inet manual
mtu 9000

auto bond0
iface bond0 inet manual
bond-slaves eth6 eth7
bond-miimon 100
bond-mode active-backup
bond-primary eth6
mtu 9000

auto vmbr0
iface vmbr0 inet static
address X.X.X.X/X
gateway X.X.X.X
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000

auto vmbr450
iface vmbr450 inet manual
bridge-ports vlan450
bridge-stp off
bridge-fd 0
mtu 9000

auto vlan450
iface vlan450 inet manual
mtu 9000
vlan-raw-device bond0

In summary, I aim to achieve a state where the interfaces consist of vmbr0, directly connected to bond0, handling all traffic, and VLAN interfaces, also associated with bond0, receiving frames with specific tags. Based on the VLAN interfaces, VMBR interfaces are created, for example, for VLAN 450, there is vmbr450.

On vm with vlan interfaces inside (when configure networking inside VM's communication doesn't work - vlan450): :
1712222272626.png


On another vm, when vmbr450 is connected directly (when configure networking inside VM's communication works fine):
1712222299088.png



Regards
Tom
Click to expand...
I personally prefer OpenvSwitch for networking, specially in a vlan heavy environment. It makes life a lot easier.

If you are trying to configure your network where each VM will handle the tagging inside, then the simplest configuration is the following:
Code: 
auto vmbr0
iface vmbr0 inet manual
    bridge_ports bond0
    bridge_stp off
    bridge_fd 0
With this configuration, simply assign vmbr0 to the VM and and configure network interface inside the VM with vlan tag. When handling large number of vlans, this is the simplest method as it does not require creating seperate virtual bridge for each vlan. This is also known as Guest configured vlan.
 
@wahmed I have this configuration

Code: 
auto vmbr0
iface vmbr0 inet static
address X.X.X.X/X
gateway X.X.X.X
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000

auto vmbr450
iface vmbr450 inet manual
bridge-ports vlan450
bridge-stp off
bridge-fd 0
mtu 9000

auto vlan450
iface vlan450 inet manual
mtu 9000
vlan-raw-device bond0

But I have a problem when I using vmbr0 for VM and configure vlan450 inside VM's. Below my VM's config which use vmbr0.

Code: 
network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      addresses:
        - X.X.X.X/X
      nameservers:
        addresses: [8.8.8.8]
      routes:
        - to: default
          via: X.X.X.X

  vlans:
    vlan450:
      id: 450
      link: eth0
      addresses: [10.X.X.X/24]

In this case communication via vlan450 doesn't work.
But when I add vmbr450 to antother VM's I can ping hosts in vlan450.
 
@gawron737 If you want to configure vlan tag inside a VM, you cannot assign a bridge already configured as vlan. The bridge needs to be free from any vlan configuration at the host level. So, in your case, the network configuration on Proxmox will look something like this:
Code: 
auto vmbr0
iface vmbr0 inet static
address X.X.X.X/X
gateway X.X.X.X
bridge-ports bond0
bridge-stp off
bridge-fd 0
mtu 9000
No extra bridge is required to pass vlan into the VM. You will configure the Vlan inside the VM itself whether by tagging or by creating an interface such as eth0.450 inside VM. The main bridge vmbr0 will pass all tagged packets into the VM. The VM will only accept the vlan it is configured for.

Keep in mind that this is not a secured practice as the VM will potentially see all tagged packets in your environment.
 
Without vlan-aware and when I delete vmbr450 and vlan450 communication works but I need two ways of communication for vlan450 (over vmbr0 and over dedicated vmbr450). How to resolve this problem because this config doesn't work:

Code: 
auto vmbr0
iface vmbr0 inet static
address X.X.X.X/X
gateway X.X.X.X
bridge-ports bond0
bridge-stp off
bridge-fd 0
mtu 9000

auto vmbr450
iface vmbr450 inet manual
bridge-ports vlan450
bridge-stp off
bridge-fd 0
mtu 9000

auto vlan450
iface vlan450 inet manual
mtu 9000
vlan-raw-device bond0

Do you understand my problem?
 
If you are trying to configure a VM with a dedicated bridge as well, then do the following configuration. Do not create a separate vlan450:
Code: 
auto vmbr450
iface vmbr450 inet manual
bridge-ports bond0
bridge-vlan-aware yes
bridge-vids 450
bridge-stp off
bridge-fd 0
mtu 9000

auto vlan450
iface vlan450 inet manual
mtu 9000
vlan-raw-device bond0
 
When I tried use this one:
Code: 
auto vmbr450
iface vmbr450 inet manual
bridge-ports bond0.450
bridge-stp off
bridge-fd 0
vlan-raw-device bond0
mtu 9000

I got output:
Code: 
ifup vmbr450
error: misconfig..? bond0 bridge port is enslaved to multiple interfaces ['bond0.450', 'vmbr0', 'vmbr450']
error: could not determine vlanid
 
Last edited:
I find only solution like this:

Code: 
auto vmbr0.450
iface vmbr0.450 inet manual
    mtu 9000

auto vmbr450
iface vmbr450 inet manual
    bridge-ports vmbr0.450
    bridge-stp off
    bridge-fd 0
    mtu 9000

Do you know any problems with this solution or this solution is quite good? Is this solution is supported by Proxmox or not recommended?
 
gawron737 said:
I find only solution like this:

Code: 
auto vmbr0.450
iface vmbr0.450 inet manual
    mtu 9000

auto vmbr450
iface vmbr450 inet manual
    bridge-ports vmbr0.450
    bridge-stp off
    bridge-fd 0
    mtu 9000

Do you know any problems with this solution or this solution is quite good? Is this solution is supported by Proxmox or not recommended?
Click to expand...
Yes, you can try this configuration as well. The code will look like this:
Code: 
auto vmbr0
iface vmbr0 inet static
      address X.X.X.X/X
      gateway X.X.X.X
      bridge-ports bond0
      bridge-stp off
      bridge-fd 0
      bridge-vlan-aware yes
      bridge-vids 2-4094
      mtu 9000

auto vmbr0.450
iface vmbr0.450 inet manual
      bridge-stp off
      bridge-fd 0
      mtu 9000
This is not Proxmox specific issue. Proxmox simply uses standard linux bridging that comes with Debian OS. If it is still not working, please post the entire content of /etc/network/interfaces. It is always a best practice to reduce abstraction layer as much as possible on default linux bridge. So the physical interface is not configured multiple times. With openvswitch this sort of configuration is easier since you can treat a virtual bridge the way you would treat a physical one.
But, standard linux bridge should work just fine for your purpose.
 
  • Like
Reactions: gawron737
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom