Where to actually put a VLAN interface

[lucius_the]

I've tried asking on ServerFault (https://serverfault.com/questions/1134269/where-to-put-vlan-interface) because it's not strictly a proxmox question. But I received no answers, only a suggestion to post on this forum, so I'm reposting here.

I have 2 x 1 Gbps network interfaces. On top of them I have Linux bond (LACP mode, but probably unimportant here). Then, on top of bond0 I have Linux bridge vmbr0, that is "VLAN aware". vmbr0 is primarily used for VM-s. A VLAN interface will be used to access the host (host management) and it needs to be in another VLAN.

Where should I put a Linux VLAN interface for host management ? There's basically 3 ways to do it, I can:

• put it on a bond0 (with bond0.VLAN_ID)
• put it on vmbr0 (with vmbr0.VLAN_ID)
• install a separate Linux VLAN interface (vlanX) with raw device that is either bond0 or vmbr0.

bond0 seemed like the place to put it on and so I did. But then, if I have a default gateway on that interface, it doesn't work - the system doesn't see or use that gateway (and has no Internet access) although I can access the host via that VLAN and ping it. Why def gw doesn't work in that case, I have no clue.

If I instead put a VLAN interface on vmbr0 then everything works, including having a default gateway there. So, sure, I can just go on with that - but. I'm here because I need some expert help to understand why putting a VLAN on one interface or the other is better/preferred/etc and why it doesn't work if I put it directly on bond0 ?

Anyways, where would you put it ? What place should be preferred (and why).

Thanks in advance

This is my config:

auto lo
iface lo inet loopback

auto enp11s0
iface enp11s0 inet manual
        mtu 9000
#Onboard port 1 (only IMM)

iface enx02215e6bcbe5 inet manual
        mtu 9000

auto enp16s0f0
iface enp16s0f0 inet manual
        mtu 9000
#Intel PCIEx port 1

auto enp16s0f1
iface enp16s0f1 inet manual
        mtu 9000
#Intel PCIEx port 2

auto enp21s0
iface enp21s0 inet manual
        mtu 9000
#Onboard port 2

auto bond0
iface bond0 inet manual
        bond-slaves enp16s0f0 enp16s0f1
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer3+4
        mtu 9000
#LACP link

auto vmbr0
iface vmbr0 inet static
        address 10.10.40.42/23
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 9000

auto vmbr0.23
iface vmbr0.23 inet static
        address 10.10.23.2/24
        gateway 10.10.23.254
        mtu 8000
#host management

 

[spirit]

if you use vlan-aware bridge, use vmbr0.X (your config is correct)

if you don't use vlan-aware, use bond0.X

 

[lucius_the]

Thank you for your answer. I figured as much, but could you explain in more details - for instance why bond0.23 is not the way to go (when bridge is vlan-aware as you mention) ?

My question is here actually in an effort to gain a deeper understanding of this. I know this works, I want to understand why the other doesn't work. Also what's with a separate VLAN interface, why is that option there (if it's pointess/redundant) ?

 

[spirit]

some reasons I have in mind:

- vlan 23 will catched by bond0.23 , so will never reach vmbr0 (if you need vm also using this vlan)
- some nics (depend of the model), are doing vlan offloading/filtering, and mixing both tagging on bond0.x and vlan-aware bridge (as your vms will use tags on the vmbr) is not working

 

[lucius_the]

That makes sense, thank you !

One thing I noticed (I will be using VLAN 23 only to access host management, for such use case it's even better if vmbr0 can't have access to it, so that's good for me) is that when I make a bond0.23 interface - if I assign a default gateway to that interface (IP config is the same as with vmbr0.23) -> default gateway is not being used. This puzzles me a bit.

Another question I have is (provided both bond0.23 and vmbr0.23 would be available as a choice, if both would work with a def gw) are there any performance reasons in choosing one or the other ? Does a bridge add some measurable overhead ?