Ver 7 bond problem

na7kr · Jul 9, 2021

Was working in old version:

Code:

● networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2021-07-08 14:58:25 PDT; 14s ago
       Docs: man:interfaces(5)
    Process: 10743 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
   Main PID: 10743 (code=exited, status=1/FAILURE)
        CPU: 111ms

Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up ibp3s0
Jul 08 14:58:25 na7kr-pve-2 ifup[10761]: No iface stanza found for master bond0
Jul 08 14:58:25 na7kr-pve-2 ifup[10758]: run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up ibp3s0d1
Jul 08 14:58:25 na7kr-pve-2 ifup[10767]: /etc/network/if-pre-up.d/ifenslave: 39: echo: echo: I/O error
Jul 08 14:58:25 na7kr-pve-2 ifup[10778]: RTNETLINK answers: File exists
Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up bond0
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: networking.service: Failed with result 'exit-code'.
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: Failed to start Raise network interfaces.

ip -d addr


1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 46 maxmtu 9000 
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.d4:ae:52:8a:53:8c designated_root 8000.d4:ae:52:8a:53:8c hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 
    altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether d4:ae:52:8a:53:8d brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 46 maxmtu 9000 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 
    altname enp1s0f1
4: ibp3s0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520 
    ipoib pkey 0xffff mode datagram umcast 0000 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
5: ibp3s0d1: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc pfifo_fast master bond0 state DOWN group default qlen 256
    link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520 
    ipoib pkey 0xffff mode datagram umcast 0000 
    bond_slave state BACKUP mii_status DOWN link_failure_count 0 perm_hwaddr 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c queue_id 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.d4:ae:52:8a:53:8c designated_root 8000.d4:ae:52:8a:53:8c root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer   47.71 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 
    inet 10.70.1.13/22 brd 10.70.3.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::d6ae:52ff:fe8a:538c/64 scope link 
       valid_lft forever preferred_lft forever
7: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/infiniband e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 
    bond mode active-backup miimon 100 updelay 0 downdelay 0 peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none arp_all_targets any primary_reselect always fail_over_mac active xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 gso_max_segs 65535 
    inet 10.80.1.13/24 brd 10.80.1.255 scope global bond0
       valid_lft forever preferred_lft forever

cat /etc/network/interfaces


auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.70.1.13/22
        hwaddress d4:ae:52:8a:53:8c
        gateway 10.70.0.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

iface eno2 inet manual
       hwaddress d4:ae:52:8a:53:8d

auto ibp3s0
iface ibp3s0 inet manual
    bond-master bond0
    bond-primary ibp1s0
    bond-mode active-backup
    mtu 9000
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b

auto ibp3s0d1
iface ibp3s0d1 inet manual 
  bond-master bond0
    bond-primary ibp1s0
    bond-mode active-backup
    mtu 9000
    hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c

# Define master
auto bond0
iface bond0 inet static
    bond-slaves ibp3s0d1
    bond-primary ibp3s0
    bond-mode active-backup
    bond-miimon 100
    address 10.80.1.13
    hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00     
    netmask 255.255.255.0
    mtu 9000

ph0x · Jul 9, 2021

The definition of your bond seems inconsistent.
ibp3s0 and ibp3s0d1 have bond0 as master but bond0 only has ibp3s0d1 as slave.
Additionally, the first two interfaces have a non-existent ibp1s0 as primary whereas bond0 has ibp3s0 as primary.

na7kr · Jul 9, 2021

I have updated to :


auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.70.1.13/22
        hwaddress d4:ae:52:8a:53:8c
        gateway 10.70.0.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

iface eno2 inet manual
       hwaddress d4:ae:52:8a:53:8d

auto ibp3s0
iface ibp3s0 inet manual
    bond-master bond0
    bond-primary ibp3s0
    bond-mode active-backup
    mtu 9000
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b

auto ibp3s0d1
iface ibp3s0d1 inet manual 
    bond-master bond0
    bond-primary ibp3s0
    bond-mode active-backup
    mtu 9000
    hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c

# Define master
auto bond0
iface bond0 inet static
    bond-slaves none
    bond-primary ibp3s0
    bond-mode active-backup
    bond-miimon 100
    address 10.80.1.13
    hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00     
    netmask 255.255.255.0
    mtu 9000

looks same as https://wiki.debian.org/Bonding
configuration - Example 2
Where is the error?

ph0x · Jul 9, 2021

Looks okay now I would say. What does cat /proc/net/bonding/bond0 say?

czechsys · Jul 9, 2021

Test this - remove all "auto iface_name" and use it only for vmbr0 and bond0.

na7kr · Jul 9, 2021

cat: /proc/net/bonding/bond0: No such file or directory


auto lo
iface lo inet loopback

auto vmbr0
iface vmbr0 inet static
        address 10.70.1.13/22
        hwaddress d4:ae:52:8a:53:8c
        gateway 10.70.0.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

# Define master
auto bond0
iface bond0 inet static
    slaves ibp3s0 ibp3s0d1
    bond-primary ibp3s0
    bond-mode active-backup
    bond-miimon 100
    address 10.80.1.13
    hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00
    netmask 255.255.255.0
    gateway 10.80.1.1
    mtu 9000

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d4:ae:52:8a:53:8d brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
4: ibp3s0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: ibp3s0d1: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff
inet 10.70.1.13/22 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:fe8a:538c/64 scope link
valid_lft forever preferred_lft forever

ph0x · Jul 9, 2021

The suggestion was to only remove the lines beginning with »auto«, not the whole interfaces.

na7kr · Jul 9, 2021

cat: /proc/net/bonding/bond0: No such file or directory


auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.70.1.13/22
        hwaddress d4:ae:52:8a:53:8c
        gateway 10.70.0.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0

iface eno2 inet manual
#       hwaddress d4:ae:52:8a:53:8d


iface ibp3s0 inet static
    bond-master bond0
    bond-primary ibp3s0 ibp3s0d1
    bond-mode active-backup
    mtu 9000
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b


iface ibp3s0d1 inet static
    bond-master bond0
    bond-primary ibp3s0 ibp3s0d1
    bond-mode active-backup
    mtu 9000
    hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c

# Define master
auto bond0
iface bond0 inet static
    slaves ibp3s0 ibp3s0d1
    bond-primary ibp3s0
    bond-mode active-backup
    bond-miimon 100
    address 10.80.1.13
    hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00
    netmask 255.255.255.0
    gateway 10.80.1.1
    mtu 9000

na7kr · Jul 9, 2021

Remove MTU and bond0 up.

Found this

Debian Bug report logs - #722496
mtu not always set properly on bond/vlan interface

ph0x · Jul 10, 2021

Please try this one:

Code:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
  address 10.70.1.13/22
  hwaddress d4:ae:52:8a:53:8c
  ### Either a gateway in vmbr0 or in bond0, not in both! ###
  gateway 10.70.0.1
  ############################################
  bridge_ports eno1
  bridge_stp off
  bridge_fd 0

iface ibp3s0 inet manual
  hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b
  mtu 9000

iface ibp3s0d1 inet manual
  hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c
  mtu 9000

# Define master
auto bond0
iface bond0 inet static
  bond-slaves ibp3s0 ibp3s0d1
  bond-primary ibp3s0
  bond-mode active-backup
  bond-miimon 100
  address 10.80.1.13/24
  hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00
  ### Either a gateway in vmbr0 or in bond0, not in both! ###
  gateway 10.80.1.1
  ############################################
  mtu 9000

spirit · Jul 10, 2021

Code:

auto ibp3s0
iface ibp3s0 inet manual
    mtu 9000
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b

auto ibp3s0d1
iface ibp3s0d1 inet manual
    mtu 9000
    hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c

static->manual: (no ip address on interface)
remove bond options (should be define on bond interface only)

ph0x · Jul 10, 2021

All this is set correctly if you define it via the GUI, btw.

na7kr · Jul 12, 2021

The GUI does not work with my InfiniBand cards so not a option.
Will try today above.

na7kr · Jul 12, 2021

Same does not work still down down

ph0x · Jul 12, 2021

Please post your /etc/network/interfaces once more. And /proc/net/bonding/bond0 is still not existing?

na7kr · Jul 13, 2021


auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.70.1.12/22
        gateway 10.70.0.1
        bridge_ports eno1
        bridge_stp off
        bridge_fd 0
        hwaddress 78:2b:cb:6d:dc:23

iface eno2 inet manual


iface ibp3s0 inet manual
   mtu 9000

iface ibp3s0d1 inet manual
  mtu 9000



# Define master
auto bond0
iface bond0 inet static
    bond-slaves ibp3s0d1 ibp3s0
    bond-primary ibp3s0
    bond-mode active-backup
    bond-miimon 100
    address 10.80.1.12
    netmask 255.255.255.0
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:5b:43
    mtu 9000

Above if from another host.

ph0x · Jul 13, 2021

Now we're getting closer. I missed the part where it said »infiniband«. And as the error message says, it's related to the interface type.
Unfortunately, I can't help you with infiniband interfaces. With normal ethernet interfaces your config would be okay.
I suggest to open a new thread which makes it clear from the beginning that you want to bond infiniband interfaces.

Search

Search

Ver 7 bond problem

na7kr

Member

ph0x

Renowned Member

na7kr

Member

ph0x

Renowned Member

czechsys

Renowned Member

na7kr

Member

ph0x

Renowned Member

na7kr

Member

na7kr

Member

Debian Bug report logs - #722496
mtu not always set properly on bond/vlan interface

ph0x

Renowned Member

spirit

Distinguished Member

ph0x

Renowned Member

na7kr

Member

na7kr

Member

ph0x

Renowned Member

na7kr

Member

ph0x

Renowned Member

Ver 7 bond problem

Member

Renowned Member

Member

Renowned Member

Renowned Member

Member

Renowned Member

Member

Member

Debian Bug report logs - #722496 mtu not always set properly on bond/vlan interface​

Renowned Member

Distinguished Member

Renowned Member

Member

​

Member

Renowned Member

Member

Renowned Member

Debian Bug report logs - #722496
mtu not always set properly on bond/vlan interface