[SOLVED] Network best practice?

[baron164]

I am new to Proxmox and I'm setting up my first server. So far the only thing I'm confused on is the networking. I currently have a linux bridge (vmbr0) and a linux bond (bond0). Do I need both or can I get away with just having the bond? My host has 4x 1gb nics and I want to bond all four of them. I'm not able to add the network gateway to both so I assume I can only have one or the other. I included a screenshot of what I have so far. Ideally I would have all four nics on the bond and call it a day. I'm just not sure if I need the bridge or not and what would happen if I removed it.

 

[shanreich]

A bridge is needed for sharing an interface with multiple guests - so yes, you probably will need one. You can find a good introduction on the fundamental building blocks of linux networking here [1]. That might help you.

Having two physical interfaces on the same bridge (as shown on your screenshot) is almost never what you want. If you want to add multiple interfaces to a bridge, you should bond them beforehand otherwise you run into the risk of loops and other undesired behaviour.

For bonds we usually recommend using either LACP (802.3ad) or active-backup. balance-rr, like LACP, needs dedicated switch support. You need to make sure that the switch is properly configured before using it. Additionally it has the downside that packets can arrive out-of-order leading to TCP retransmits. The downside of LACP is that you can only use one interface per connection which means that the maximum speed for one connection is 1 GBit.

Is this a single server or do you plan on clustering? If you plan to create a cluster it might make sense to use the multiple NICs for different networks: corosync, management, vm traffic, ...

[1] https://developers.redhat.com/blog/...on-to-linux-interfaces-for-virtual-networking

 

[baron164]

Ok I think I understand this now. I intend to configure LACP eventually, but for right now I'm just trying to ensure that I can configure a bond properly. So Active-Backup will work for the time being. I reviewed the proxmox documentation located here and saw this:

Another possibility it to use the bond directly as bridge port. This can be used to make the guest network fault-tolerant.
Example: Use a bond as bridge port

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
address 10.10.10.2/24
gateway 10.10.10.1
bridge-ports bond0
bridge-stp off
bridge-fd 0

So it looks like I just need to assign the Linux Bridge to use the bond instead of the individual nics.

 

[shanreich]

So it looks like I just need to assign the Linux Bridge to use the bond instead of the individual nics.

Yes, exactly

 

[baron164]

Thanks for the help

 

[jdancer]

I use this config in production. YMMV.

    # Configure Dell rNDC X540/I350 Quad NIC card with 10GbE active and 1GbE as backup
    # VLAN 10 = Management network traffic
    # VLAN 20 30 40 = VM network traffic
    #
    auto lo
    iface lo inet loopback
   
    iface eno1 inet manual
    #10GbE
   
    iface eno2 inet manual
    #10GbE

    iface eno3 inet manual
    #1GbE

    iface eno4 inet manual
    #1GbE
   
    auto bond0
    iface bond0 inet manual
        bond-slaves eno1 eno3
        bond-miimon 100
        bond-mode active-backup
        bond-primary eno1
    #Mgmt Bond Interface
   
    auto vmbr0
    iface vmbr0 inet manual
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 10
    #Mgmt VLAN Bridge Interface
   
    auto vmbr0.10
    iface vmbr0.10 inet static
        address 192.168.1.10/24
        gateway 192.168.1.1
    #Mgmt Web Interface
   
    auto bond1
    iface bond1 inet manual
        bond-slaves eno2 eno4
        bond-miimon 100
        bond-mode active-backup
        bond-primary eno2
    #VM Bond Interface
   
    auto vmbr1
    iface vmbr1 inet manual
        bridge-ports bond1
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 20 30 40
    #VM VLAN Bridge Interface