Ver 7 bond problem

[na7kr]

Was working in old version:

● networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2021-07-08 14:58:25 PDT; 14s ago
       Docs: man:interfaces(5)
    Process: 10743 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
   Main PID: 10743 (code=exited, status=1/FAILURE)
        CPU: 111ms

Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up ibp3s0
Jul 08 14:58:25 na7kr-pve-2 ifup[10761]: No iface stanza found for master bond0
Jul 08 14:58:25 na7kr-pve-2 ifup[10758]: run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up ibp3s0d1
Jul 08 14:58:25 na7kr-pve-2 ifup[10767]: /etc/network/if-pre-up.d/ifenslave: 39: echo: echo: I/O error
Jul 08 14:58:25 na7kr-pve-2 ifup[10778]: RTNETLINK answers: File exists
Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up bond0
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: networking.service: Failed with result 'exit-code'.
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: Failed to start Raise network interfaces.

ip -d addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 46 maxmtu 9000 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.d4:ae:52:8a:53:8c designated_root 8000.d4:ae:52:8a:53:8c hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp1s0f0 3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether d4:ae:52:8a:53:8d brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 46 maxmtu 9000 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp1s0f1 4: ibp3s0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256 link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520 ipoib pkey 0xffff mode datagram umcast 0000 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 5: ibp3s0d1: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc pfifo_fast master bond0 state DOWN group default qlen 256 link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520 ipoib pkey 0xffff mode datagram umcast 0000 bond_slave state BACKUP mii_status DOWN link_failure_count 0 perm_hwaddr 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c queue_id 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.d4:ae:52:8a:53:8c designated_root 8000.d4:ae:52:8a:53:8c root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 47.71 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 10.70.1.13/22 brd 10.70.3.255 scope global vmbr0 valid_lft forever preferred_lft forever inet6 fe80::d6ae:52ff:fe8a:538c/64 scope link valid_lft forever preferred_lft forever 7: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/infiniband e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bond mode active-backup miimon 100 updelay 0 downdelay 0 peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none arp_all_targets any primary_reselect always fail_over_mac active xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 gso_max_segs 65535 inet 10.80.1.13/24 brd 10.80.1.255 scope global bond0 valid_lft forever preferred_lft forever



cat /etc/network/interfaces
auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 iface eno2 inet manual hwaddress d4:ae:52:8a:53:8d auto ibp3s0 iface ibp3s0 inet manual bond-master bond0 bond-primary ibp1s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b auto ibp3s0d1 iface ibp3s0d1 inet manual bond-master bond0 bond-primary ibp1s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c # Define master auto bond0 iface bond0 inet static bond-slaves ibp3s0d1 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 mtu 9000

 

[ph0x]

The definition of your bond seems inconsistent.
ibp3s0 and ibp3s0d1 have bond0 as master but bond0 only has ibp3s0d1 as slave.
Additionally, the first two interfaces have a non-existent ibp1s0 as primary whereas bond0 has ibp3s0 as primary.

 

[na7kr]

I have updated to :
auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 iface eno2 inet manual hwaddress d4:ae:52:8a:53:8d auto ibp3s0 iface ibp3s0 inet manual bond-master bond0 bond-primary ibp3s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b auto ibp3s0d1 iface ibp3s0d1 inet manual bond-master bond0 bond-primary ibp3s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c # Define master auto bond0 iface bond0 inet static bond-slaves none bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 mtu 9000

looks same as https://wiki.debian.org/Bonding
configuration - Example 2
Where is the error?

 

[ph0x]

Looks okay now I would say. What does cat /proc/net/bonding/bond0 say?

 

[czechsys]

Test this - remove all "auto iface_name" and use it only for vmbr0 and bond0.

 

[na7kr]

cat: /proc/net/bonding/bond0: No such file or directory
auto lo iface lo inet loopback auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 # Define master auto bond0 iface bond0 inet static slaves ibp3s0 ibp3s0d1 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 gateway 10.80.1.1 mtu 9000

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d4:ae:52:8a:53:8d brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
4: ibp3s0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: ibp3s0d1: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff
inet 10.70.1.13/22 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:fe8a:538c/64 scope link
valid_lft forever preferred_lft forever

 

[ph0x]

The suggestion was to only remove the lines beginning with »auto«, not the whole interfaces.

 

[na7kr]

cat: /proc/net/bonding/bond0: No such file or directory

auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 iface eno2 inet manual # hwaddress d4:ae:52:8a:53:8d iface ibp3s0 inet static bond-master bond0 bond-primary ibp3s0 ibp3s0d1 bond-mode active-backup mtu 9000 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b iface ibp3s0d1 inet static bond-master bond0 bond-primary ibp3s0 ibp3s0d1 bond-mode active-backup mtu 9000 hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c # Define master auto bond0 iface bond0 inet static slaves ibp3s0 ibp3s0d1 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 gateway 10.80.1.1 mtu 9000

 

[na7kr]

Remove MTU and bond0 up.

Found this

Debian Bug report logs - #722496
mtu not always set properly on bond/vlan interface​

 

[ph0x]

Please try this one:

auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
  address 10.70.1.13/22
  hwaddress d4:ae:52:8a:53:8c
  ### Either a gateway in vmbr0 or in bond0, not in both! ###
  gateway 10.70.0.1
  ############################################
  bridge_ports eno1
  bridge_stp off
  bridge_fd 0

iface ibp3s0 inet manual
  hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b
  mtu 9000

iface ibp3s0d1 inet manual
  hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c
  mtu 9000

# Define master
auto bond0
iface bond0 inet static
  bond-slaves ibp3s0 ibp3s0d1
  bond-primary ibp3s0
  bond-mode active-backup
  bond-miimon 100
  address 10.80.1.13/24
  hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00
  ### Either a gateway in vmbr0 or in bond0, not in both! ###
  gateway 10.80.1.1
  ############################################
  mtu 9000

 

[spirit]

auto ibp3s0
iface ibp3s0 inet manual
    mtu 9000
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b

auto ibp3s0d1
iface ibp3s0d1 inet manual
    mtu 9000
    hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c

static->manual: (no ip address on interface)
remove bond options (should be define on bond interface only)

 

[ph0x]

All this is set correctly if you define it via the GUI, btw.

 

[na7kr]

The GUI does not work with my InfiniBand cards so not a option.
Will try today above.

​

 

[na7kr]

Same does not work still down down

 

[ph0x]

Please post your /etc/network/interfaces once more. And /proc/net/bonding/bond0 is still not existing?

 

[na7kr]

auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.12/22 gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 hwaddress 78:2b:cb:6d:dc:23 iface eno2 inet manual iface ibp3s0 inet manual mtu 9000 iface ibp3s0d1 inet manual mtu 9000 # Define master auto bond0 iface bond0 inet static bond-slaves ibp3s0d1 ibp3s0 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.12 netmask 255.255.255.0 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:5b:43 mtu 9000

Above if from another host.

 

[ph0x]

Now we're getting closer. I missed the part where it said »infiniband«. And as the error message says, it's related to the interface type.
Unfortunately, I can't help you with infiniband interfaces. With normal ethernet interfaces your config would be okay.
I suggest to open a new thread which makes it clear from the beginning that you want to bond infiniband interfaces.