Ver 7 bond problem

na7kr

New Member
May 28, 2021
11
1
3
57
Was working in old version:

Code:
● networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2021-07-08 14:58:25 PDT; 14s ago
       Docs: man:interfaces(5)
    Process: 10743 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
   Main PID: 10743 (code=exited, status=1/FAILURE)
        CPU: 111ms

Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up ibp3s0
Jul 08 14:58:25 na7kr-pve-2 ifup[10761]: No iface stanza found for master bond0
Jul 08 14:58:25 na7kr-pve-2 ifup[10758]: run-parts: /etc/network/if-pre-up.d/ifenslave exited with return code 1
Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up ibp3s0d1
Jul 08 14:58:25 na7kr-pve-2 ifup[10767]: /etc/network/if-pre-up.d/ifenslave: 39: echo: echo: I/O error
Jul 08 14:58:25 na7kr-pve-2 ifup[10778]: RTNETLINK answers: File exists
Jul 08 14:58:25 na7kr-pve-2 ifup[10743]: ifup: failed to bring up bond0
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: networking.service: Failed with result 'exit-code'.
Jul 08 14:58:25 na7kr-pve-2 systemd[1]: Failed to start Raise network interfaces.

ip -d addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000 link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 46 maxmtu 9000 bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.d4:ae:52:8a:53:8c designated_root 8000.d4:ae:52:8a:53:8c hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on mcast_to_unicast off neigh_suppress off group_fwd_mask 0 group_fwd_mask_str 0x0 vlan_tunnel off isolated off numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp1s0f0 3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether d4:ae:52:8a:53:8d brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 46 maxmtu 9000 numtxqueues 8 numrxqueues 8 gso_max_size 65536 gso_max_segs 65535 altname enp1s0f1 4: ibp3s0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256 link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520 ipoib pkey 0xffff mode datagram umcast 0000 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 5: ibp3s0d1: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc pfifo_fast master bond0 state DOWN group default qlen 256 link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520 ipoib pkey 0xffff mode datagram umcast 0000 bond_slave state BACKUP mii_status DOWN link_failure_count 0 perm_hwaddr 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c queue_id 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bridge forward_delay 0 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.d4:ae:52:8a:53:8c designated_root 8000.d4:ae:52:8a:53:8c root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer 0.00 tcn_timer 0.00 topology_change_timer 0.00 gc_timer 47.71 vlan_default_pvid 1 vlan_stats_enabled 0 vlan_stats_per_port 0 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 16 mcast_hash_max 4096 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3124 mcast_stats_enabled 0 mcast_igmp_version 2 mcast_mld_version 1 nf_call_iptables 0 nf_call_ip6tables 0 nf_call_arptables 0 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 inet 10.70.1.13/22 brd 10.70.3.255 scope global vmbr0 valid_lft forever preferred_lft forever inet6 fe80::d6ae:52ff:fe8a:538c/64 scope link valid_lft forever preferred_lft forever 7: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/infiniband e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 bond mode active-backup miimon 100 updelay 0 downdelay 0 peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none arp_all_targets any primary_reselect always fail_over_mac active xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 gso_max_segs 65535 inet 10.80.1.13/24 brd 10.80.1.255 scope global bond0 valid_lft forever preferred_lft forever



cat /etc/network/interfaces
auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 iface eno2 inet manual hwaddress d4:ae:52:8a:53:8d auto ibp3s0 iface ibp3s0 inet manual bond-master bond0 bond-primary ibp1s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b auto ibp3s0d1 iface ibp3s0d1 inet manual bond-master bond0 bond-primary ibp1s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c # Define master auto bond0 iface bond0 inet static bond-slaves ibp3s0d1 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 mtu 9000
 

ph0x

Renowned Member
Jul 5, 2020
1,326
212
68
/dev/null
The definition of your bond seems inconsistent.
ibp3s0 and ibp3s0d1 have bond0 as master but bond0 only has ibp3s0d1 as slave.
Additionally, the first two interfaces have a non-existent ibp1s0 as primary whereas bond0 has ibp3s0 as primary.
 

na7kr

New Member
May 28, 2021
11
1
3
57
I have updated to :
auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 iface eno2 inet manual hwaddress d4:ae:52:8a:53:8d auto ibp3s0 iface ibp3s0 inet manual bond-master bond0 bond-primary ibp3s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b auto ibp3s0d1 iface ibp3s0d1 inet manual bond-master bond0 bond-primary ibp3s0 bond-mode active-backup mtu 9000 hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c # Define master auto bond0 iface bond0 inet static bond-slaves none bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 mtu 9000

looks same as https://wiki.debian.org/Bonding
configuration - Example 2
Where is the error?
 

czechsys

Well-Known Member
Nov 18, 2015
331
28
48
Test this - remove all "auto iface_name" and use it only for vmbr0 and bond0.
 

na7kr

New Member
May 28, 2021
11
1
3
57
cat: /proc/net/bonding/bond0: No such file or directory
auto lo iface lo inet loopback auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 # Define master auto bond0 iface bond0 inet static slaves ibp3s0 ibp3s0d1 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 gateway 10.80.1.1 mtu 9000

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether d4:ae:52:8a:53:8d brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
4: ibp3s0: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: ibp3s0d1: <BROADCAST,MULTICAST> mtu 4092 qdisc noop state DOWN group default qlen 256
link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether d4:ae:52:8a:53:8c brd ff:ff:ff:ff:ff:ff
inet 10.70.1.13/22 scope global vmbr0
valid_lft forever preferred_lft forever
inet6 fe80::d6ae:52ff:fe8a:538c/64 scope link
valid_lft forever preferred_lft forever
 

ph0x

Renowned Member
Jul 5, 2020
1,326
212
68
/dev/null
The suggestion was to only remove the lines beginning with »auto«, not the whole interfaces. :)
 

na7kr

New Member
May 28, 2021
11
1
3
57
cat: /proc/net/bonding/bond0: No such file or directory

auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.13/22 hwaddress d4:ae:52:8a:53:8c gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 iface eno2 inet manual # hwaddress d4:ae:52:8a:53:8d iface ibp3s0 inet static bond-master bond0 bond-primary ibp3s0 ibp3s0d1 bond-mode active-backup mtu 9000 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b iface ibp3s0d1 inet static bond-master bond0 bond-primary ibp3s0 ibp3s0d1 bond-mode active-backup mtu 9000 hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c # Define master auto bond0 iface bond0 inet static slaves ibp3s0 ibp3s0d1 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.13 hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00 netmask 255.255.255.0 gateway 10.80.1.1 mtu 9000
 
Last edited:

na7kr

New Member
May 28, 2021
11
1
3
57
Remove MTU and bond0 up.

Found this

Debian Bug report logs - #722496
mtu not always set properly on bond/vlan interface​

 
Last edited:
  • Like
Reactions: David Herselman

ph0x

Renowned Member
Jul 5, 2020
1,326
212
68
/dev/null
Please try this one:
Code:
auto lo
iface lo inet loopback

iface eno1 inet manual

iface eno2 inet manual

auto vmbr0
iface vmbr0 inet static
  address 10.70.1.13/22
  hwaddress d4:ae:52:8a:53:8c
  ### Either a gateway in vmbr0 or in bond0, not in both! ###
  gateway 10.70.0.1
  ############################################
  bridge_ports eno1
  bridge_stp off
  bridge_fd 0

iface ibp3s0 inet manual
  hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b
  mtu 9000

iface ibp3s0d1 inet manual
  hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c
  mtu 9000

# Define master
auto bond0
iface bond0 inet static
  bond-slaves ibp3s0 ibp3s0d1
  bond-primary ibp3s0
  bond-mode active-backup
  bond-miimon 100
  address 10.80.1.13/24
  hwaddress e2:76:a1:b3:3f:e1:00:00:00:00:00:00:00:00:00:00:00:00:00:00
  ### Either a gateway in vmbr0 or in bond0, not in both! ###
  gateway 10.80.1.1
  ############################################
  mtu 9000
 
Last edited:

spirit

Famous Member
Apr 2, 2010
5,768
673
133
www.odiso.com
Code:
auto ibp3s0
iface ibp3s0 inet manual
    mtu 9000
    hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7b

auto ibp3s0d1
iface ibp3s0d1 inet manual
    mtu 9000
    hwaddress 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:61:7c


static->manual: (no ip address on interface)
remove bond options (should be define on bond interface only)
 

na7kr

New Member
May 28, 2021
11
1
3
57
The GUI does not work with my InfiniBand cards so not a option.
Will try today above.

 

ph0x

Renowned Member
Jul 5, 2020
1,326
212
68
/dev/null
Please post your /etc/network/interfaces once more. And /proc/net/bonding/bond0 is still not existing?
 

na7kr

New Member
May 28, 2021
11
1
3
57
Untitled.png

auto lo iface lo inet loopback iface eno1 inet manual auto vmbr0 iface vmbr0 inet static address 10.70.1.12/22 gateway 10.70.0.1 bridge_ports eno1 bridge_stp off bridge_fd 0 hwaddress 78:2b:cb:6d:dc:23 iface eno2 inet manual iface ibp3s0 inet manual mtu 9000 iface ibp3s0d1 inet manual mtu 9000 # Define master auto bond0 iface bond0 inet static bond-slaves ibp3s0d1 ibp3s0 bond-primary ibp3s0 bond-mode active-backup bond-miimon 100 address 10.80.1.12 netmask 255.255.255.0 hwaddress 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:5b:43 mtu 9000

Above if from another host.
 

ph0x

Renowned Member
Jul 5, 2020
1,326
212
68
/dev/null
Now we're getting closer. I missed the part where it said »infiniband«. And as the error message says, it's related to the interface type.
Unfortunately, I can't help you with infiniband interfaces. With normal ethernet interfaces your config would be okay.
I suggest to open a new thread which makes it clear from the beginning that you want to bond infiniband interfaces.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!