LACP + Vlan + Disaster scenario

[HellrazorX]

Hi everyone,

I'm running a particular scenario here

I'm using three nics in total.

enp9s0f0
enp9s0f1

Are bond to LACP, slaves to Vmbr1, vlan aware
Should run on a 66 Vlan

enp3s0

slave to vmbr0, vlan aware
should run on primairy gateway

---

The reason for this setup is to, mainly run on LACP and leave enp3s0 disconnected from the network,
In case of having my main router dying or a switch, i'll be quickly able to put back the network up using the basic interface using the base gateway.

I read a lot of documentation and tried to combine example scenarios but i'm getting confused. Some suggested parameters can be typed down in ''interface'' config file, but aren't valid to GUI interface so this is why I'm asking.

Here is my current config file

auto lo iface lo inet loopback iface enp3s0 inet manual auto enp9s0f0 iface enp9s0f0 inet manual auto enp9s0f1 iface enp9s0f1 inet manual auto vmbr0 iface vmbr0 inet static address 192.168.6.66/24 gateway 192.168.6.1 bridge-ports enp3s0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 auto bond0 iface bond0 inet manual address 192.168.66.66/24 bond-slaves enp9s0f0 enp9s0f1 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2+3 auto vmbr1 iface vmbr1 inet static bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094

Vmbr1 does not seem to be able to connect to the network.


Any Idea? What am I horribly doing wrong?


Thank you

 

[HellrazorX]

Some progress here...
Was able to effectively run a second NIC on a VLAN, passing through other VLANS. Now. I'm LAGG doesn't work properly. here is the current config.

auto lo
iface lo inet loopback

iface enp3s0 inet manual

auto enp9s0f0
iface enp9s0f0 inet manual

auto enp9s0f1
iface enp9s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.6.66/24
        gateway 192.168.6.1
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto bond0
iface bond0 inet manual
      bond-slaves enp9s0f0 enp9s0f1
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3

iface bond0.66 inet manual

auto vmbr1v66
iface vmbr1v66 inet static
        address 192.168.66.66/24
        gateway 192.168.66.1
        bridge-ports bond0.66
        bridge-stp off
        bridge-fd 0

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

Ironically, bond0 works only if ONE cable is plugged to the nic.
From a hardware perspective, the config worked on a simple bond, pretty sure I'm messing up the configuration with one line somewhere. Close but not there still.

Thanks if you have any idea,

Hx

 

[spirit]

you need to remove

"
bridge-vlan-aware yes
bridge-vids 2-4094
"
from vmbr1 as you use non vlan aware setup with "vmbr1v66".

iface bond0.66 inet manual

auto vmbr1v66
iface vmbr1v66 inet static
        address 192.168.66.66/24
        gateway 192.168.66.1
        bridge-ports bond0.66
        bridge-stp off
        bridge-fd 0


auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

or if you need vlan-aware, and a management ip on vlan66, you need to do

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
       bridge-vlan-aware yes
       bridge-vids 2-4094

auto vmbr1.66
iface vmbr1.66 inet static
        address 192.168.66.66/24
        gateway 192.168.66.1

but don't mix both

 

[HellrazorX]

you need to remove

"
bridge-vlan-aware yes
bridge-vids 2-4094
"
from vmbr1 as you use non vlan aware setup with "vmbr1v66".

iface bond0.66 inet manual

auto vmbr1v66
iface vmbr1v66 inet static
        address 192.168.66.66/24
        gateway 192.168.66.1
        bridge-ports bond0.66
        bridge-stp off
        bridge-fd 0


auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0

or if you need vlan-aware, and a management ip on vlan66, you need to do

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
       bridge-vlan-aware yes
       bridge-vids 2-4094

auto vmbr1.66
iface vmbr1.66 inet static
        address 192.168.66.66/24
        gateway 192.168.66.1

but don't mix both

Thanks for your kind reply. I tried second example and I simply don't get any net access to the server VIA bond0 anymore.

Is there a line missing?
Just to be clear.

vmbr0 is fine (vlan aware, running on base lan)

vmbr1 needs to be:
from the bond0
vlan aware
running on vlan 66

I'm sorry if this seems like taking my hand for walk but it seems like there are some subtilities where I didn't find answers for yet:

• Static vs manual vs automatic (why and where)
• .5 or v5 for identifying vlans
• Addesses and gateways are sometime defined over the adapter, sometime over the bridge.

Thank you

here is another attempt, like another trial and error I made, I can ACCESS the pve through the 66 adapter but, connection seems very unstable when both cables are plugged in:

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

iface enp3s0 inet manual

auto enp9s0f0
iface enp9s0f0 inet manual

auto enp9s0f1
iface enp9s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.6.66/24
        gateway 192.168.6.1
        bridge-ports enp3s0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto bond0
iface bond0 inet manual
      bond-slaves enp9s0f0 enp9s0f1
      bond-miimon 100
      bond-mode 802.3ad
      bond-xmit-hash-policy layer2+3


auto vmbr1.66
iface vmbr1.66 inet static
        address 192.168.66.66/24
        gateway 192.168.66.1

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

 

[HellrazorX]

Still doing experiments and the more I play with it, the more I begin to suspect Proxmox has simply a problem dealing with a second Bridge.
Why can't it be set on two gateways?

The only way I sucessfully was able to experience a truly working bond was when I set the bond as the ONLY connection to the server (192.168.66.66/24) For everything.

I tried everything using Linux bonds, should I start considering using OVS? have I reached the limit of Linux interfaces are facing?
I'd really like to finish this very initial setup because it's the step blocking the path to my cluster config.

Thanks...

 

[spirit]

Static vs manual

static is when on a have an ip on the interface
manual is an interface without ip

vs automatic (why and where)

auto ... , is to have the interface start at boot (you can set auto everywhere without problem)

• .5 or v5 for identifying vlans

vmbr.5, is when you have a vlanaware bridge, to define an internal vlan interface to add an ip on the bridge. (the vlan tag is done inside the bridge)

vmbrv5, is a proxmox way, to define a non-vlaware bridge. in this bridge, a physical interface. eth0.5 is in bridge-ports, so the vlan tag is done when the packet is going out through the eth0.5.

• Addesses and gateways are sometime defined over the adapter, sometime over the bridge.

Address can't be define on an interface, is the interface is inside " bridge-ports" ... In this case, the ip need to be set on the bridge directly.
It's a linux kernel limitation.

Why can't it be set on two gateways?

because this is how network is working.
the gateway simply said: "to join any network (0.0.0.0.0) go through this gateway".
They are not relationship to the src ip.

for example, you have this network:

192.168.0.10 (adminpc)---------192.168.0.1(router)----192.168.66.1-----192.168.66.66 (proxmox node vlanX)
10.0.0.10 (adminpc)---------10.0.0.1(another router)----192.168.6.1-----192.168.6.6 (proxmox node vlan Y)

if an admin pc is calling 1 of the proxmox node ip, the packet will arrive to the server without any problem,
but for the reply, the default gw to 0.0.0.0 (to 192.168.0.10 or 10.0.0.10 or any other network will be use).

the only way to manage this, is to create more granular routes on proxmox nodes with:
"post-up ip route add 10.0.0.0/24 gw 192.168.6.1"
"post-up ip route add 192.168.0.0/24 gw 192.168.66.1"

Hope it's help