Need help with broadcast bonded LAN setup

special_case · Sep 15, 2022

I want to use multiple NICs on my Proxmox machine as a switch with routing across the multiple LAN NICs and to the single WAN NIC. I have them set up as a broadcast bond (see details below), with enp2s0 through enp6s0 all bonded. I can see all the Proxmox VMs and the WAN from LAN devices, but I cannot see LAN devices on other interfaces that are part of the bond. Is broadcast bond not the way to do this?

Details below. Note the proxmox host is 192.168.0.11 and the OPNSense VM which is my LAN gateway and WAN/LAN bridge is 192.168.0.1.

> cat /etc/network/interfaces:

Code:

auto lo
iface lo inet loopback

iface enp1s0 inet manual

auto vmbr0
iface vmbr0 inet dhcp
    bridge-ports enp1s0
    bridge-stp off
    bridge-fd 0

auto enp2s0
iface enp2s0 inet manual

auto enp3s0
iface enp3s0 inet manual

auto enp4s0
iface enp4s0 inet manual

auto enp5s0
iface enp5s0 inet manual

auto enp6s0
iface enp6s0 inet manual

auto bond0
iface bond0 inet manual
    bond-slaves enp2s0 enp3s0 enp4s0 enp5s0 enp6s0
    bond-miimon 100
    bond-mode broadcast

auto vmbr1
iface vmbr1 inet static
    address 192.168.0.11/24
    gateway 192.168.0.1
    bridge-ports bond0
    bridge-stp off
    bridge-fd 0

> ip a (with addresses censored):

Code:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
3: enp2s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
4: enp3s0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx permaddr xx:xx:xx:xx:xx:xx
5: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx permaddr xx:xx:xx:xx:xx:xx
6: enp5s0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx permaddr xx:xx:xx:xx:xx:xx
7: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx permaddr xx:xx:xx:xx:xx:xx
8: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
    inet xxx.xxx.xxx.xxx/26 brd xxx.xxx.xxx.xxx scope global dynamic vmbr0
       valid_lft 4193sec preferred_lft 4193sec
    inet6 xxxx::xxx:xxxx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
10: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
    inet 192.168.0.11/24 scope global vmbr1
       valid_lft forever preferred_lft forever
    inet6 xxxx::xxx:xxxx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
11: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i0 state UNKNOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
12: fwbr100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
13: fwpr100p0@fwln100i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
14: fwln100i0@fwpr100p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
15: tap100i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr100i1 state UNKNOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
16: fwbr100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
17: fwpr100p1@fwln100i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
18: fwln100i1@fwpr100p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr100i1 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
19: tap102i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr1 state UNKNOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
20: tap104i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr104i0 state UNKNOWN group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
21: fwbr104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
22: fwpr104p0@fwln104i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx
23: fwln104i0@fwpr104p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr104i0 state UP group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd xx:xx:xx:xx:xx:xx

ip route:

Code:

default via xxx.xxx.xxx.xxx dev vmbr0
192.168.0.0/24 dev vmbr1 proto kernel scope link src 192.168.0.11
xxx.xxx.xxx.xxx/26 dev vmbr0 proto kernel scope link src xxx.xxx.xxx.xxx

vesalius · Sep 16, 2022

From the one full example I can find of using broadcast in the proxmox Docs: https://pve.proxmox.com/wiki/Full_Mesh_Network_for_Ceph_Server#Command-Line

looks like you might want to try just slaves and not bond-slaves;

Code:

auto bond0
iface bond0 inet manual
    slaves enp2s0 enp3s0 enp4s0 enp5s0 enp6s0
    bond-miimon 100
    bond-mode broadcast

Search

Search

Need help with broadcast bonded LAN setup

special_case

New Member

vesalius

Renowned Member