Confused on bonds, interfaces..

[markf1301]

Hello all.
Doesn't take much, but here's what I have.
pve -


From what I am trying to understand -
eno1 and eno2 are active and in the bridge of vmbro.
Both interfaces are active and 1 is being used for mgmt, and the other is where my vlan's are running out of.
The box0 is running off of 2 different interfaces, enp1s0f0 and enp1sof1 - and configured in a active/backup solution.

pvepbr - 2nd box in cluster

Only 1 interface is active - due to issues on the switch, which i can't get into.
Whenever I 'migrate' from 1 server to another, and there is a vlan associated with it from PVE server, i have to stop/migrate/adjust the vlan from what it should be, to the 'default' vlan on pvepbr.

What I'm trying to do is essentially the same on pvepbr as on pve - create a bridge which can accept the different vlans on it (specified on the hardware/vlan setting as well as (somehow) create a bond whereby the managment port is dedicated to 1 of the interfaces on pvepbr, and the 2nd interface is handling all the different vlans.

Since this is in production currently, how can i do this?
I can, if need be - migrate all servers from the 2nd box, pvepbr, over to pve and make changes as needed.
I was unsure if i needed to create the bond first, and associated it with the 2 interfaces (eno1 and eno2) but as eno2 wasn't connected to the switch - I was unsure.
If I do this, then connect the 2nd interface to the switch - what will happen to connectivity to the box? Will the 'bond' now start advertising as the new ip, currently associated with the server?
After this is done, then presumably I can create the bridge and add the bond0 to it. ???

How does the system know which address or interface that the vlans are running on, vs the mgmt port?

thanks in advance, and can provide more info if needed.

 

[alexskysilk]

eno1 and eno2 are active and in the bridge of vmbro.

this doesn't work like you think. In effect, this creates a loop which will either have one interface blocked by STP, or the whole switch would be shut down. In other words, this is a poor substitute for an active/passive bond.

The box0 is running off of 2 different interfaces, enp1s0f0 and enp1sof1 - and configured in a active/backup solution.

That is a more appropriate method. If LACP is available to you (check your switch) you should use that instead. you get all the benefits PLUS all (well, most) of the bandwidth.

I find the PVE gui for network management limiting. If you post the content of your /etc/network/interfaces file we can work on getting exactly where you want to go.

 

[markf1301]

Here's the info on both the pve and pvepbr as well as the interface status.

on another note, i need to change the default gateway on both boxes as i'm replacing the current 'gw' with another address. can this be done just by changing this file, or are there other files that need to be 'touched' as well?

thanks.

 

[alexskysilk]

here's what to do.

create a new file named /etc/network/interfaces.new with the following content:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual
    
auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode active-backup
# If LACP is an option:
#       bond-mode 802.3ad
#       bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 172.16.x.x/24
        gateway 172.16.x.x
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

source /etc/network/interfaces.d/*

reboot to apply.

 

[markf1301]

here's what to do.

create a new file named /etc/network/interfaces.new with the following content:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual
   
auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode active-backup
# If LACP is an option:
#       bond-mode 802.3ad
#       bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        address 172.16.x.x/24
        gateway 172.16.x.x
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

source /etc/network/interfaces.d/*

reboot to apply.

A couple of assumptions.
1. this is regarding the pvepbr box which currently only has 1 interface connected to the switch. Correct?
2. this will create the 'bond' or bond0 for both of those interfaces.
3. i'm unsure about lacp on the switch. I know the current Hp switch has this configuration but am also moving the host-servers over to my new environment - which is a Fortigate environment with managed fortigate switches. i'll have to review the docs on the fortigate side before i can determine how to set the lacp option, by port to get this to work.

Once this is done, then presumably i plug both interfaces from the server over into the corresponding switch ports - and the 'bond0' will come up, but only 1 port will be active and the other is in 'backup/standby' mode. correct?

 

[alexskysilk]

1. this is regarding the pvepbr box which currently only has 1 interface connected to the switch. Correct?

correct

2. this will create the 'bond' or bond0 for both of those interfaces.

correct

3. i'm unsure about lacp on the switch. I know the current Hp switch has this configuration but am also moving the host-servers over to my new environment - which is a Fortigate environment with managed fortigate switches. i'll have to review the docs on the fortigate side before i can determine how to set the lacp option, by port to get this to work.

You really should. if both eno1 and eno2 go to the same switch in an active/passive configuration it doesn't actually provide any utility over using just a single nic.

Once this is done, then presumably i plug both interfaces from the server over into the corresponding switch ports - and the 'bond0' will come up, but only 1 port will be active and the other is in 'backup/standby' mode. correct?

correct. this configuration works on any switch as long as both ports are on the same vlan.

 

[markf1301]

A couple of assumptions.
1. this is regarding the pvepbr box which currently only has 1 interface connected to the switch. Correct?
2. this will create the 'bond' or bond0 for both of those interfaces.
3. i'm unsure about lacp on the switch. I know the current Hp switch has this configuration but am also moving the host-servers over to my new environment - which is a Fortigate environment with managed fortigate switches. i'll have to review the docs on the fortigate side before i can determine how to set the lacp option, by port to get this to work.

Once this is done, then presumably i plug both interfaces from the server over into the corresponding switch ports - and the 'bond0' will come up, but only 1 port will be active and the other is in 'backup/standby' mode. correct?

thanks on the lacp thing. I presume you referenced the file with .new so that I can create the new file with the .new extension and then cp interfaces interfaces.old (to keep what I have) and then cp interfaces.new to interfaces so that I'm replacing the current existing file. Then plug into the switch, and reboot the box. Correct? Sorry for the steps, but in this new environment I just need to understand all the processes, and what happens 'next'. Worse case, I can always plug back into the old switch and revert the interface file. I surmise.

 

[alexskysilk]

presume you referenced the file with .new so that I can create the new file with the .new extension and then cp interfaces interfaces.old (to keep what I have) and then cp interfaces.new to interfaces so that I'm replacing the current existing file.

Thats actually not necessary. PVE will automatically deploy interfaces.new on boot if present. just create the file and reboot. Incidentally, this configuration will work the same as the original one in any case, since only one of the bond slaves needs to be plugged in to work.

 

[markf1301]

Thats actually not necessary. PVE will automatically deploy interfaces.new on boot if present. just create the file and reboot. Incidentally, this configuration will work the same as the original one in any case, since only one of the bond slaves needs to be plugged in to work.

Thanks for the clarification. I didn't know that would happen.
I'll make the change tomorrow and see what happens.
Just found out the Fortigate 148 switches will do LACP, but not MCLAP between the 2 different boxes so I'll make that change as well and update back.

 

[markf1301]

Thanks for the clarification. I didn't know that would happen.
I'll make the change tomorrow and see what happens.
Just found out the Fortigate 148 switches will do LACP, but not MCLAP between the 2 different boxes so I'll make that change as well and update back.

I tried the 'update' to the interfaces and it didn't work. no ip connectivity anywhere, except itself.
I have created a port trunk on my fortigate switch, configured as 'active/active'. I tried with 'default vlan' of 100 and no default with vlan100 allowed. no change. reverted back till i can figure this out. I also tried removing the 2 comments about the lacp-active?, and rebooted inbetween.
no go...no collect $200. Anything else as a suggestion?
I'm going to ping the fortigate group to confirm active/active is set as 802.1x and update this thread back.

 

[alexskysilk]

pay attention to your bond stanza.

If you're trying to connect it to a lagg, use the lacp config. If you're connecting to access ports, use the active/backup configuration.

In either case, your switch should have a flow analysis tool (I dont know how fortigate os commands differ for switches, but for routers it would be

diagnose debug enable
diagnose debug flow filter addr (address of the pve interface)
diagnose debug flow trace start 50

 

[markf1301]

pay attention to your bond stanza.

If you're trying to connect it to a lagg, use the lacp config. If you're connecting to access ports, use the active/backup configuration.

In either case, your switch should have a flow analysis tool (I dont know how fortigate os commands differ for switches, but for routers it would be

diagnose debug enable
diagnose debug flow filter addr (address of the pve interface)
diagnose debug flow trace start 50

i presume you are referring to removing or using 1, or the other.
I have 3 choices on the trunk - static, passive lacp, and active lacp. it was set as 'active lacp' when i did my testing.

in the config - which should be left to make this work?
I'd thought I'd tried all the combinations but might not have.

 

[alexskysilk]

Not entirely sure what passive LACP is, but for your purposes its not a valid option. Consult your switch documentation on what it expects on the other end, but in my experience they all respect layer2+3 (I could be wrong.)

so to reiterate:

auto bond0
iface bond0 inet manual
        bond-slaves eno1 eno2
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

 

[markf1301]

Ok. some success.
Here's the current interfaces config below -
Strange thing - i cannot ping the gw address now. any other address on the same network - 172.16.x.x are reachable as well as the internet, from the host server.
vm's that are running - default vlan (100) are about to reach the internet, and other others on the same network - but not their gw address either, which is the .254 address.
Other address in the same 172.16.x.x network can ping the host server and reach all the vm's w/o issue.

the fortigate switch is configured (only way to get the interface up) with the trunk mode in the VL100 network.
I haven't tested unplugging 1 interface to see what happens but will do that shortly.


auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
# bond-mode active-backup
# If LACP is an option:
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
address 172.16.x.221/24
gateway 172.16.x.254
bridge-ports bond0
bridge-stp off

 

[markf1301]

DOLT !!!!!!

forgot to enable ping on the gw.
working now.
dual nic is working now.

YEA !!!!!

 

[markf1301]

Ok - so 1 down.
what's wrong with this config? Same switch.
Tried with and w/o the additional commands - bold/commented out.
The switch 'default vlan' is the same 172.16.x.x network - so no 'tagging' involved.

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

auto enp1s0f0
iface enp1s0f0 inet manual

auto enp1s0f1
iface enp1s0f1 inet manual

auto bond0
iface bond0 inet manual
bond-slaves enp1s0f0 enp1s0f1
bond-miimon 100
bond-mode active-backup
bond-primary enp1s0f0
#10g-bond

auto vmbr0
iface vmbr0 inet static
address 172.16.x.x/24
gateway 172.16.x.x
bridge-ports eno1 eno2
bridge-stp off
bridge-fd 0
# bond-mode 802.3ad
#bond-xmit-hash-policy layer2+3
#management bond

auto vmbr1
iface vmbr1 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
#vlan networks

 

[alexskysilk]

you have vmbr0 using two uplinks directly and bridge stp turned off. Either turn it on, make sure the uplinks do NOT go to the same switch fabric, or make sure STP is turned on upstream from you.

not really sure what you're after- if you can describe what each of your 4 interfaces is meant to do I can help with the configuration.

 

[markf1301]

sure - probably should have started with that and to be sure I under the configuration of proxmox, and my current config..
there are 3 things to networking - in this example.

ethernet interfaces - en01, eno2, enp1s0f0, and enp1s0f1
bonds are made up of multiple interfaces. in my config, i'm creating a bond (vmbr0) which consists of eno1 and eno2.
the 2nd bond is made of of enp1s0f0 and enp1s0f1 named as bond0.
also note that enp1s0f0 is set as the primary and the other should 'come up' if the primary is down. correct?

The switch is a fortigate and is managing the trunk where vmbr0 is connected to. - it can do either static, active, or something called passive but i have it configured as 'active/active'.
stp, or spanning tree is active or on for the trunk port.

What i'm trying to do is use vmbr0 as the management bond (vmbr0) so that i have 2 interfaces in case 1 fails and keep the 'tagged' traffic, on the other bond/trunk - vmbr0.

i haven't started or tried to get bond0, the enpxxx interfaces up yet - much less tried to pass 'tagged' traffic off of that bond.

If this works then I'd be using 2 interfaces for each of the trunks with all 4 interfaces active, or in active/fail situation regarding the ethernet ports.

Does this make sense?

 

[alexskysilk]

bonds are made up of multiple interfaces. in my config, i'm creating a bond (vmbr0) which consists of eno1 and eno2.

vmbr= linux bridge. its not a bond. think of it as a switch.

you have one bond defined, bond0, and it is using enp1s0f0 and enp1s0f1 as its captive interfaces (slaves.)

if you want to use eno1 and eno2 in a bond, then that's what you should tell the networking layer to do, eg:

auto bond1
iface bond1 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode ...

and then assign the bridge to use that interface as an uplink.

 

[guruevi]

Just to clarify:
- you have physical NIC
- a bond creates a single NIC out of multiple (1 or more) physical NIC. You can create bond0, bond1, … in a bond all physical NIC set the same MAC address
- - LACP is a special type of a bond that negotiates between the NICs and the switches, active side sends packets to negotiate which links are participating, passive side listens for config packets
- - active/passive will just activate one link until the link dies, then the other takes over
- - there are load balancing and other types, if the switch doesn’t know what to do with the same MAC address on 2 ports, it should send packets destined for that MAC on BOTH ports simultaneously, or shut down the port(s) depending on its configuration.

Once you have a bond, abstract it away as a single NIC and start over
Each VM is like a real machine with (typically) a single NIC, abstract like that and start from that
Each bridge is a virtual switch, abstract it away like that
The OS hosting bonds or bridges does not have a ‘NIC’ into either (it will move the traffic, just not accept it for its own purposes) unless your host gives it an IP address.

- you can pass a NIC to a single VM, at which point that NIC passes traffic to the VM
- you can pass a NIC to a bridge, each interface will register its MAC addresses and the switch will send data according, or if not found/broadcast packet, it will forward to the default port which is (typically) the NIC(s) you configure in /etc/interfaces

Now you can make a diagram and see what makes sense. Follow the traffic. What happens when you connect two switches to each other over multiple independent uplink interfaces, if you don’t have a topology aware setup like STP, a broadcast packet on that switch will be sent to the uplinks, now the other switch sees 2 broadcast packets for the same MAC, if the other switch is unaware of the topology (STP) the 2 packets get sent back becoming 4 packets etc until the entire thing crashes.

Similarly, what happens when you have multiple links each to different switches (eg. switch0 -> bond0 -> bridge0 -> bond1 -> switch1), well now switch0 can just talk to switch1 through bridge0. You can see how powerful/dangerous this becomes.

You have (or want to have) 2 bonds of 2 interfaces each, once the bond is setup, they become just 2 NIC, provided your switch ‘understands’ the bond (active/passive or LACP) you can now connect that to the physical switch. Now you have 2 NIC, one management, one ‘customer’, what do they need to do, multiple VM/container - then you’ll need a switch (bridge) otherwise just use them as regular interfaces passed to the host or a VM/container.

If you can avoid VLAN at this point, I would suggest it, but again, abstract it, VLAN just tell a switch what traffic to segregate. You can have a trunk (can pass traffic for multiple VLAN) or an access port (all traffic will always be on a specific VLAN). All components need to be aware of VLAN, otherwise they will just ignore those VLAN tags, meaning, if you have a switch, that is set up over a trunk port and it is not VLAN aware, it could accept traffic for all those VLAN and forward them to the ‘default’ VLAN or any of its own ports.