ceph and bond LACP failed after upgrade from 7 to 8

Jun 25, 2022
68
7
13
oday i upgraded one server to version 8, but ceph failed to start, two of the OSd in the node is down, after a search for few hours, what i found is that the bond LACP (balanced tcs) is not working after the upgrade, there is 4 bond interface in the node, out of these one is on mesh network for ceph in broadcast protocol, which is working and the management interface in lacp is working, rest of the two bond is down, coz of that ceph cluster network in a lacp bond is not pinging and the ceph cluster is gone down, i also found that in network tab, "Apply Configuration" is also not working, ifupdown2 is installed and working in version 7, but after upgrade all r dead....... any help
 
yes i found out the interface name of the extra card are changed...... i am attaching the interface

# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

auto lo
iface lo inet loopback

auto eno12399np0
iface eno12399np0 inet manual

auto enp177s0f2
iface enp177s0f2 inet manual

auto enp177s0f3
iface enp177s0f3 inet manual

auto eno8303
iface eno8303 inet manual

auto eno8403
iface eno8403 inet manual

auto eno12409np1
iface eno12409np1 inet manual

auto enp178s0f0np0
iface enp178s0f0np0 inet manual

auto enp178s0f1np1
iface enp178s0f1np1 inet manual

auto enp177s0f0
iface enp177s0f0 inet manual

auto enp177s0f1
iface enp177s0f1 inet manual

iface ens4f0np0 inet manual

iface ens4f1np1 inet manual

iface ens5f0 inet manual

iface ens5f1 inet manual

iface ens5f2 inet manual

iface ens5f3 inet manual

auto mgnt
iface mgnt inet static
address 192.168.137.112/24
gateway 192.168.137.1
ovs_type OVSIntPort
ovs_bridge vmbr0

auto data
iface data inet static
address 192.168.137.119/24
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_mtu 9000

auto pvecluster1
iface pvecluster1 inet static
address 192.168.30.112/24
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options tag=30
#pvecluster-1

auto pvecluster2
iface pvecluster2 inet static
address 10.10.30.112/24
ovs_type OVSIntPort
ovs_bridge vmbr1
ovs_mtu 9000
ovs_options tag=30
#pvecluster-2

auto pvecluster3
iface pvecluster3 inet static
address 192.168.30.12/24
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_mtu 9000
ovs_options tag=30
#pvecluster-3

auto pveceph
iface pveceph inet static
address 192.168.20.112/24
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_mtu 9000
ovs_options tag=20
#ceph-cluster

auto migration
iface migration inet static
address 10.10.50.112/24
ovs_type OVSIntPort
ovs_bridge vmbr2
ovs_mtu 9000
ovs_options tag=50
#HA-Migration

auto ups
iface ups inet static
address 192.168.50.112/24
ovs_type OVSIntPort
ovs_bridge vmbr3

auto wan
iface wan inet manual
ovs_type OVSIntPort
ovs_bridge vmbr3
ovs_options tag=10
#BSNL

auto airtel
iface airtel inet manual
ovs_type OVSIntPort
ovs_bridge vmbr3
#Airtel

auto nas
iface nas inet static
address 192.168.0.112/24
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_mtu 9000

auto bond0
iface bond0 inet manual
ovs_bonds eno12399np0 eno12409np1
ovs_type OVSBond
ovs_bridge vmbr0
ovs_mtu 9000
ovs_options bond_mode=balance-tcp lacp=active

auto bond1
iface bond1 inet manual
ovs_bonds enp177s0f0 enp177s0f1
ovs_type OVSBond
ovs_bridge vmbr1
ovs_mtu 9000
ovs_options lacp=active bond_mode=balance-tcp

auto bond2
iface bond2 inet manual
ovs_bonds enp178s0f0np0 enp178s0f1np1
ovs_type OVSBond
ovs_bridge vmbr2
ovs_mtu 9000
ovs_options lacp=active bond_mode=balance-tcp

auto bond3
iface bond3 inet static
address 10.10.20.112/24
bond-slaves enp177s0f2 enp177s0f3
bond-miimon 100
bond-mode broadcast
mtu 9000
#ceph-public

auto bond4
iface bond4 inet manual
ovs_bonds eno8303 eno8403
ovs_type OVSBond
ovs_bridge vmbr3
ovs_mtu 9000
ovs_options lacp=active bond_mode=balance-tcp
#1G

auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports bond0 mgnt pvecluster1 nas
ovs_mtu 9000

auto vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports bond1 data pvecluster2
ovs_mtu 9000

auto vmbr2
iface vmbr2 inet manual
ovs_type OVSBridge
ovs_ports bond2 pvecluster3 pveceph migration
ovs_mtu 9000

auto vmbr3
iface vmbr3 inet manual
ovs_type OVSBridge
ovs_ports bond4 ups wan airtel
ovs_mtu 9000

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: eno8303: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
link/ether ec:2a:72:30:12:40 brd ff:ff:ff:ff:ff:ff
altname enp4s0f0
inet6 fe80::ee2a:72ff:fe30:1240/64 scope link
valid_lft forever preferred_lft forever
3: eno12399np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 84:16:0c:71:1d:b0 brd ff:ff:ff:ff:ff:ff
altname enp51s0f0np0
inet6 fe80::8616:cff:fe71:1db0/64 scope link
valid_lft forever preferred_lft forever
4: eno8403: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
link/ether ec:2a:72:30:12:41 brd ff:ff:ff:ff:ff:ff
altname enp4s0f1
inet6 fe80::ee2a:72ff:fe30:1241/64 scope link
valid_lft forever preferred_lft forever
5: eno12409np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 84:16:0c:71:1d:b1 brd ff:ff:ff:ff:ff:ff
altname enp51s0f1np1
inet6 fe80::8616:cff:fe71:1db1/64 scope link
valid_lft forever preferred_lft forever
6: ens5f0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000
link/ether b4:96:91:f7:e6:0c brd ff:ff:ff:ff:ff:ff
altname enp177s0f0
7: ens5f1: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000
link/ether b4:96:91:f7:e6:0d brd ff:ff:ff:ff:ff:ff
altname enp177s0f1
8: ens4f0np0: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000
link/ether 5c:6f:69:b2:7b:10 brd ff:ff:ff:ff:ff:ff
altname enp178s0f0np0
9: ens4f1np1: <BROADCAST,MULTICAST> mtu 9000 qdisc noop state DOWN group default qlen 1000
link/ether 5c:6f:69:b2:7b:11 brd ff:ff:ff:ff:ff:ff
altname enp178s0f1np1
10: ens5f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond3 state UP group default qlen 1000
link/ether b4:96:91:f7:e6:0e brd ff:ff:ff:ff:ff:ff
altname enp177s0f2
11: ens5f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 9000 qdisc mq master bond3 state UP group default qlen 1000
link/ether b4:96:91:f7:e6:0e brd ff:ff:ff:ff:ff:ff permaddr b4:96:91:f7:e6:0f
altname enp177s0f3
12: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 22:1b:53:63:0d:b6 brd ff:ff:ff:ff:ff:ff
13: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 84:16:0c:71:1d:b0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::c84b:68ff:fed9:e440/64 scope link
valid_lft forever preferred_lft forever
14: mgnt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether e2:65:b9:9b:e9:c8 brd ff:ff:ff:ff:ff:ff
inet 192.168.137.112/24 scope global mgnt
valid_lft forever preferred_lft forever
inet6 fe80::e065:b9ff:fe9b:e9c8/64 scope link
valid_lft forever preferred_lft forever
15: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether d6:9d:84:d9:00:42 brd ff:ff:ff:ff:ff:ff
inet6 fe80::d49d:84ff:fed9:42/64 scope link
valid_lft forever preferred_lft forever
16: data: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether d6:df:98:16:e8:79 brd ff:ff:ff:ff:ff:ff
inet 192.168.137.119/24 scope global data
valid_lft forever preferred_lft forever
inet6 fe80::d4df:98ff:fe16:e879/64 scope link
valid_lft forever preferred_lft forever
17: pvecluster1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 86:71:49:81:e9:ba brd ff:ff:ff:ff:ff:ff
inet 192.168.30.112/24 scope global pvecluster1
valid_lft forever preferred_lft forever
inet6 fe80::8471:49ff:fe81:e9ba/64 scope link
valid_lft forever preferred_lft forever
18: pvecluster2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 4e:53:e7:f6:41:be brd ff:ff:ff:ff:ff:ff
inet 10.10.30.112/24 scope global pvecluster2
valid_lft forever preferred_lft forever
inet6 fe80::4c53:e7ff:fef6:41be/64 scope link
valid_lft forever preferred_lft forever
19: vmbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 0a:fe:1b:7b:0d:4c brd ff:ff:ff:ff:ff:ff
inet6 fe80::8fe:1bff:fe7b:d4c/64 scope link
valid_lft forever preferred_lft forever
20: pvecluster3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 6e:26:dc:28:d2:51 brd ff:ff:ff:ff:ff:ff
inet 192.168.30.12/24 scope global pvecluster3
valid_lft forever preferred_lft forever
inet6 fe80::6c26:dcff:fe28:d251/64 scope link
valid_lft forever preferred_lft forever
21: pveceph: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 3a:e2:72:f8:f2:94 brd ff:ff:ff:ff:ff:ff
inet 192.168.20.112/24 scope global pveceph
valid_lft forever preferred_lft forever
inet6 fe80::38e2:72ff:fef8:f294/64 scope link
valid_lft forever preferred_lft forever
22: migration: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether d2:ad:ce:d4:24:7e brd ff:ff:ff:ff:ff:ff
inet 10.10.50.112/24 scope global migration
valid_lft forever preferred_lft forever
inet6 fe80::d0ad:ceff:fed4:247e/64 scope link
valid_lft forever preferred_lft forever
23: vmbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether ec:2a:72:30:12:40 brd ff:ff:ff:ff:ff:ff
inet6 fe80::dcc0:13ff:fe18:cd45/64 scope link
valid_lft forever preferred_lft forever
24: ups: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 5a:d9:01:51:77:95 brd ff:ff:ff:ff:ff:ff
inet 192.168.50.112/24 scope global ups
valid_lft forever preferred_lft forever
inet6 fe80::58d9:1ff:fe51:7795/64 scope link
valid_lft forever preferred_lft forever
25: wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 2a:fb:e3:df:21:71 brd ff:ff:ff:ff:ff:ff
inet6 fe80::28fb:e3ff:fedf:2171/64 scope link
valid_lft forever preferred_lft forever
26: airtel: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 22:25:19:7f:72:a3 brd ff:ff:ff:ff:ff:ff
inet6 fe80::2025:19ff:fe7f:72a3/64 scope link
valid_lft forever preferred_lft forever
27: nas: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 52:78:42:9d:ef:2b brd ff:ff:ff:ff:ff:ff
inet 192.168.0.112/24 scope global nas
valid_lft forever preferred_lft forever
inet6 fe80::5078:42ff:fe9d:ef2b/64 scope link
valid_lft forever preferred_lft forever
28: bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether 7e:73:f0:4d:e4:36 brd ff:ff:ff:ff:ff:ff
inet6 fe80::7c73:f0ff:fe4d:e436/64 scope link
valid_lft forever preferred_lft forever
29: bond3: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
link/ether b4:96:91:f7:e6:0e brd ff:ff:ff:ff:ff:ff
inet 10.10.20.112/24 scope global bond3
valid_lft forever preferred_lft forever
inet6 fe80::b696:91ff:fef7:e60e/64 scope link
valid_lft forever preferred_lft forever
30: bond4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether c6:6c:49:9a:a2:be brd ff:ff:ff:ff:ff:ff
inet6 fe80::c46c:49ff:fe9a:a2be/64 scope link
valid_lft forever preferred_lft forever
 
i tried to change the interface name and click apply config ..... but it is giving error of "can not change"

also the local disk in the node filling up by itself, now at 90% , i do not keep any ISO images or any other backup in the small 100 g partition, now it is deep RED ....
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!