Management vlan on bond0

[acs_user]

Hello,
I'm trying to configure a management vlan for pve (7.1-7) with a bond and pass a vlan aware bridge to the guests. With the config below, I can reach pve but the hosts become unreachable. I had a configuration working where the pve management was on enp0s25 in the native vlan and the guests could communicate in their respective vlans from vmbr0. Could someone explain what I've missed?

iface lo inet loopback

iface enp0s25 inet manual

iface ens4f0 inet manual

iface ens4f1 inet manual

iface enp8s0f0 inet manual

iface enp8s0f1 inet manual

auto bond0
iface bond0 inet manual
      bond-slaves ens4f0 ens4f1
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

iface bond0.69 inet manual

auto vmbr0v69
iface vmbr0v69 inet static
      address 10.10.69.100/24
      gateway 10.10.69.1
      bridge-ports bond0.69
      bridge-stp off
      bridge-fd 0

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1001-2000

 

[Dunuin]

You could try it with...

iface lo inet loopback

iface enp0s25 inet manual

iface ens4f0 inet manual

iface ens4f1 inet manual

iface enp8s0f0 inet manual

iface enp8s0f1 inet manual

iface bond0 inet manual
      bond-slaves ens4f0 ens4f1
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

auto bond0.69
iface bond0.69 inet static
      address 10.10.69.100/24
      gateway 10.10.69.1

auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 1001-2000

I'm not sure if "vmbr0v69" is problematic because normally that means you want a vlan with VLANID 69 ontop of vmbr0 but with "bridge-vids 1001-2000" you don't allow vmbr0 to route VLANID 69 tagged packets.

 

[aghadjip]

Hey, So ive got this working, there are a few pieces of the puzzle, i dont know what you know so forgive me if i go into too much detail, please pic kout the parts that interest you.
Setup:
i have 3 proxmox hosts. management vlan is set on the 10.0.10.0/24 vlan tag of 10 subnet. We are going to call this the native subnet.

I have 2 other vlans
10.0.13.0/24 - general VMs vlan tag of 13
10.0.17.0/24 - K8 service subnets tag of 17

Networking: Your switch has to support vlan tagging. There is a concept of Native and tagged networks on a port. The native vlan is the "default vlan" meaning any traffic that doesnt have an explicit vlan tag on it will be tagged with the native vlan tag.

in my setup Native is vlan 10 (10.0.10.0/24), on my switch ive trunked vlan 10 as native, vlan 13 as tagged and vlan 17 as tagged.

my interfaces look like the following, eno1 and eno2 are 10 gig ports.

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual
    mtu 9000

auto eno2
iface eno2 inet manual
    mtu 9000

auto bond0
iface bond0 inet manual
    bond-slaves eno1 eno2
    bond-miimon 100
    bond-mode 802.3ad
    bond-xmit-hash-policy layer2+3
    mtu 9000

auto vmbr0
iface vmbr0 inet static
    address 10.0.10.40/24
    gateway 10.0.10.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0
    bridge-vlan-aware yes
    bridge-vids 2-4094
    mtu 9000

By keeping vlan 10 as native, this will make sure management port and migrations are handled over that network.

Whenever i create a new container or VM, i tag it with the appropriate vlan tag, so all my containers/VMs get tag 13 or 17

Hope that helps, make sure you do the switch bits as well.

 

[acs_user]

@Dunuin Unfortunately, that config renders everything unreachable, but I appreciate the reply.
@aghadjip I really would like to stick with tagged management. I've proven the switch config so I missing something on the Linux side. With the config below, I can reach the pve host on the management vlan and if I comment out the bridge-plan-aware yes with a native vlan configured for testing, the guests are reachable. I notice the example on the Wiki doesn't show a vlan aware bridge when using a management vlan on a bond. So this seems to be a tagging issue on vmbr0. Do you guys have any other thoughts? Thanks for the replies.

auto lo

iface lo inet loopback

iface enp0s25 inet manual

iface ens4f0 inet manual

iface ens4f1 inet manual

iface enp8s0f0 inet manual

iface enp8s0f1 inet manual

auto bond0
iface bond0 inet manual
      bond-slaves ens4f0 ens4f1
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

iface bond0.69 inet manual

auto vmbr0v69
iface vmbr0v69 inet static
      address 10.10.69.100/24
      gateway 10.10.69.1
      bridge-ports bond0.69
      bridge-stp off
      bridge-fd 0
      
auto vmbr0
iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
       # bridge-vlan-aware yes
       # bridge-vids 2-4092

 

[aghadjip]

To get a better picture, what are the other 3 interfaces connected to? The above config only uses a lagged ens4f0 and ens4f1, is that expected? It also sounds like you want the reverse of what i have.

Management on tagged vlan (looks like 10.10.69.100/24) and the tag is 69? Your first config had
bridge-vids 1001-2000
Which is outside the 69 tag, i see you corrected that in the last config

Can you tell us the config of the ethernet ports? They are set for Lagg and have vlan 69 tagged? whats the native vlan on those two?


You had this working previously you mentioned, was it only with the lagged pair? Or was one interface set as the management and had a seperate line out

 

[acs_user]

@aghadjip They are just spare NICs currently, and yes I'm trying to bond ens4f0 and ens4f1. My goal is to have the pve host managed on vlan 69 (tagged) @ 10.10.69.100/24 and each guest on it's own tagged vlan ranging from 1001-2000. Switchports are configured as 802.3ad with vlans 69,1001-2000 allowed with no native vlan (trunk port). I have this working currently on enp0s25 with the config below (no bond). I purchased (2) dual port NICs to add so I could created a bond for the pve host and use the other (2) for passing through to VMs.

auto lo
iface lo inet loopback

iface enp0s25 inet manual

iface ens4f0 inet manual

iface ens4f1 inet manual

iface enp8s0f0 inet manual

iface enp8s0f1 inet manual

auto vmbr0.69
iface vmbr0.69 inet static
     address 10.10.69.100/24
     gateway 10.10.69.1

auto vmbr0
iface vmbr0 inet manual
     bridge-ports enp0s25
     bridge-stp off
     bridge-fd 0
     bridge-vlan-aware yes
     bridge-vids 2-4094