We are using 2 bond interfaces for storage (2x10Gb) and a trunk for the VMs (2x10Gb).
We want to split the trunk for the VM in VLAN bridges so we can assign them to our VMs.
if we reload the network config we got an error:
Some other info:
Our 10Gb NIC: Broadcom BCM57412 NetXtreme-E 10Gb
Any idea what is going wrong here?
Thanks!
Erik
We want to split the trunk for the VM in VLAN bridges so we can assign them to our VMs.
Code:
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet static
address XXX.XXX.XXX.XXX/XX
gateway XXX.XXX.XXX.XXX
#node1-mgt
iface eno2 inet manual
#Cluster network
iface eno3 inet manual
iface eno4 inet manual
iface ens3f0np0 inet manual
iface ens3f1np1 inet manual
iface ens2f0np0 inet manual
iface ens2f1np1 inet manual
auto bond0
iface bond0 inet manual
bond-slaves ens2f0np0 ens3f0np0
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
bond-min-links 1
mtu 8970
#VM network (trunk)
auto bond1
iface bond1 inet static
address xxx.xxx.xxx.xxx/24
bond-slaves ens2f1np1 ens3f1np1
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
bond-min-links 1
mtu 8970
#Ceph network
auto vmbr2
iface vmbr2 inet manual
bridge-ports bond0.2
bridge-stp off
bridge-fd 0
mtu 1500
#vlan2if we reload the network config we got an error:
Code:
# ifreload -a -v
info: requesting link dump
info: requesting address dump
info: requesting netconf dump
info: loading builtin modules from ['/usr/share/ifupdown2/addons']
info: module openvswitch not loaded (module init failed: no /usr/bin/ovs-vsctl found)
info: module openvswitch_port not loaded (module init failed: no /usr/bin/ovs-vsctl found)
info: module ppp not loaded (module init failed: no /usr/bin/pon found)
info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)
info: executing /sbin/sysctl net.bridge.bridge-allow-multiple-vlans
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)
info: executing /bin/ip rule show
info: executing /bin/ip -6 rule show
info: address: using default mtu 1500
info: address: max_mtu undefined
info: executing /bin/ip addr help
info: address metric support: OK
info: module ppp not loaded (module init failed: no /usr/bin/pon found)
info: module mstpctl not loaded (module init failed: no /sbin/mstpctl found)
info: module batman_adv not loaded (module init failed: no /usr/sbin/batctl found)
info: module openvswitch_port not loaded (module init failed: no /usr/bin/ovs-vsctl found)
info: module openvswitch not loaded (module init failed: no /usr/bin/ovs-vsctl found)
info: looking for user scripts under /etc/network
info: loading scripts under /etc/network/if-pre-up.d ...
info: loading scripts under /etc/network/if-up.d ...
info: loading scripts under /etc/network/if-post-up.d ...
info: loading scripts under /etc/network/if-pre-down.d ...
info: loading scripts under /etc/network/if-down.d ...
info: loading scripts under /etc/network/if-post-down.d ...
info: 'link_master_slave' is set. slave admin state changes will be delayed till the masters admin state change.
info: processing interfaces file /etc/network/interfaces
info: reload: scheduling down on interfaces: ['vmbr4']
info: vmbr4: running ops ...
info: vmbr4: netlink: ip link set dev vmbr4 down
info: executing /etc/network/if-down.d/postfix
info: vmbr4: netlink: ip link del vmbr4
info: reload: scheduling up on interfaces: ['lo', 'eno1', 'bond1', 'bond0', 'vmbr2']
info: ens3f1np1: running ops ...
info: vrf: syncing table map to /etc/iproute2/rt_tables.d/ifupdown2_vrf_map.conf
info: vrf: dumping iproute2_vrf_map
info: {}
info: executing /sbin/sysctl net.mpls.conf.ens3f1np1.input=0
info: executing /etc/network/if-up.d/postfix
info: ens2f1np1: running ops ...
info: executing /sbin/sysctl net.mpls.conf.ens2f1np1.input=0
info: executing /etc/network/if-up.d/postfix
info: bond1: running ops ...
warning: bond1: attribute bond-min-links is set to '0'
info: bond1: already exists, no change detected
info: executing /sbin/sysctl net.mpls.conf.bond1.input=0
info: executing /etc/network/if-up.d/postfix
info: lo: running ops ...
info: executing /sbin/sysctl net.mpls.conf.lo.input=0
info: executing /etc/network/if-up.d/postfix
info: ens3f0np0: running ops ...
info: executing /sbin/sysctl net.mpls.conf.ens3f0np0.input=0
info: executing /etc/network/if-up.d/postfix
info: ens2f0np0: running ops ...
info: executing /sbin/sysctl net.mpls.conf.ens2f0np0.input=0
info: executing /etc/network/if-up.d/postfix
info: bond0: running ops ...
warning: bond0: attribute bond-min-links is set to '0'
info: bond0: already exists, no change detected
info: executing /sbin/sysctl net.mpls.conf.bond0.input=0
info: executing /etc/network/if-up.d/postfix
info: bond0.2: running ops ...
info: bond0.2: not enslaved to bridge vmbr2: ignored for now
info: executing /sbin/sysctl net.mpls.conf.bond0/2.input=0
info: executing /etc/network/if-up.d/postfix
info: vmbr2: running ops ...
info: vmbr2: bridge already exists
info: vmbr2: applying bridge settings
info: vmbr2: reset bridge-hashel to default: 4
info: reading '/sys/class/net/vmbr2/bridge/stp_state'
info: vmbr2: netlink: ip link set dev vmbr2 type bridge (with attributes)
info: writing '1' to file /proc/sys/net/ipv6/conf/bond0.2/disable_ipv6
info: executing /bin/ip -force -batch - [link set dev bond0.2 master vmbr2]
warning: vmbr2: apply bridge ports settings: cmd '/bin/ip -force -batch - [link set dev bond0.2 master vmbr2]' failed: returned 1 (RTNETLINK answers: No data available
Command failed -:1
)
info: executing /sbin/sysctl net.mpls.conf.vmbr2.input=0
info: vmbr2: bridge inherits mtu from its ports. There is no need to assign mtu on a bridge
info: executing /etc/network/if-up.d/postfix
info: eno1: running ops ...
info: executing /sbin/sysctl net.mpls.conf.eno1.input=0
info: eno1: netlink: ip addr del XXXX/64 dev eno1
info: executing /bin/ip route add default via XXX.XXX.XXX.XXX proto kernel dev eno1 onlink
info: executing /etc/network/if-up.d/postfixSome other info:
Our 10Gb NIC: Broadcom BCM57412 NetXtreme-E 10Gb
Code:
# pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.41-1-pve)
pve-manager: 6.2-4 (running version: 6.2-4/9824574a)
pve-kernel-5.4: 6.2-2
pve-kernel-helper: 6.2-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 2.0.1-1+pve8
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-1
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-2
libpve-guest-common-perl: 3.0-10
libpve-http-server-perl: 3.0-5
libpve-storage-perl: 6.1-8
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve2
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-1
pve-cluster: 6.1-8
pve-container: 3.1-6
pve-docs: 6.2-4
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-2
pve-qemu-kvm: 5.0.0-2
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-2
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1
Code:
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: bc:97:e1:76:5b:30
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 15
Partner Key: 24
Partner Mac Address: 64:64:9b:54:f5:00
Slave Interface: ens3f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: bc:97:e1:76:5b:30
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: bc:97:e1:76:5b:30
port key: 15
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 127
system mac address: 64:64:9b:54:f5:00
oper key: 24
port priority: 127
port number: 67
port state: 63
Slave Interface: ens2f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: bc:97:e1:76:c5:80
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: bc:97:e1:76:5b:30
port key: 15
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 127
system mac address: 64:64:9b:54:f5:00
oper key: 24
port priority: 127
port number: 72
port state: 63
Code:
#ip a s
...
11: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 8970 qdisc noqueue state UP group default qlen 1000
link/ether bc:97:e1:76:5b:30 brd ff:ff:ff:ff:ff:ff
inet6 fe80::be97:e1ff:fe76:5b30/64 scope link
valid_lft forever preferred_lft forever
12: bond0.2@bond0: <BROADCAST,MULTICAST> mtu 8970 qdisc noop state DOWN group default qlen 1000
link/ether bc:97:e1:76:5b:30 brd ff:ff:ff:ff:ff:ff
13: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether fa:08:b6:20:e1:e2 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f808:b6ff:fe20:e1e2/64 scope link
valid_lft forever preferred_lft forever
Code:
# brctl show
bridge name bridge id STP enabled interfaces
vmbr2 8000.000000000000 noAny idea what is going wrong here?
Thanks!
Erik
