I am trying to debug a short network interruptions (1-10 seconds) that happens to VMs in my home 3 nodes Proxmox cluster.
Those interruptions happens about once every hour so they are quite a pain to debug.
While trying to debug this issue I stumbled upon an wierd output from "brctl showmacs vmbr0".
There seems to be an extreme number of duplicates for the mac addresses that belongs to the server itself.
Output from node 1: (the first column is the uniq count)
Note the last line - 4095 entries for the local interface.
Very similar output is from node 2 and node 3 - just with different mac address.
/etc/network/interfaces (node 1):
Only node1 have eth bond configured - other nodes have single NICs as part of the vmbr0 bridge without any bonding.
ip l (node 1):
Node 1 is connected using two ethernet cables to Unifi USW-24-G1 (LACP bond).
Node 2 and 3 are connected to MikroTik RB5009UG which is then connected to the USW-24-G1.
All nodes are running up-to-date proxmox 7.2-11 (no subscription) with kernel 5.15.60-1-pve.
Is this a normal behavior or am I doing something wrong?
Any feedback is appreciated!
Those interruptions happens about once every hour so they are quite a pain to debug.
While trying to debug this issue I stumbled upon an wierd output from "brctl showmacs vmbr0".
There seems to be an extreme number of duplicates for the mac addresses that belongs to the server itself.
Output from node 1: (the first column is the uniq count)
Code:
root@wowza:~# brctl showmacs vmbr0 | uniq -c
1 port no mac addr is local? ageing timer
1 1 00:04:4b:e9:95:b9 no 24.66
1 1 00:04:4b:fb:44:f9 no 14.16
1 1 00:11:32:20:b3:04 no 5.90
1 1 00:11:32:d2:74:d9 no 165.40
1 1 00:11:32:d2:74:e0 no 165.40
1 1 00:11:32:d2:74:e1 no 0.18
1 1 00:17:88:63:ef:58 no 51.22
1 1 00:55:da:52:87:0f no 8.63
1 1 10:12:fb:27:81:ff no 169.74
1 1 10:12:fb:27:83:e3 no 169.74
1 1 10:12:fb:71:30:1d no 169.74
1 1 10:12:fb:71:30:6d no 169.73
1 1 10:12:fb:76:59:c2 no 169.74
1 1 10:12:fb:76:59:e9 no 169.74
1 1 1c:f2:9a:3e:32:a7 no 2.85
1 1 1e:87:71:30:93:a8 no 1.73
1 1 20:df:b9:48:cd:ac no 23.95
1 1 20:df:b9:55:7a:bb no 0.41
1 1 48:21:0b:2e:4d:57 no 0.45
1 1 48:21:0b:36:b4:ab no 0.36
1 1 48:21:0b:37:0e:83 no 32.35
1 1 50:14:79:1f:b8:aa no 13.93
1 1 54:60:09:d0:6c:ce no 25.20
1 1 54:60:09:f4:9a:4a no 30.18
1 1 70:03:9f:1b:d8:02 no 8.94
1 3 76:66:e1:2d:c3:fa no 0.00
1 4 76:74:13:97:1c:9c no 5.48
2 3 7e:34:36:02:4a:58 yes 0.00
1 1 7e:7b:1d:00:72:86 no 0.00
1 1 82:6f:5a:fd:ef:10 no 0.01
2 4 96:5b:45:89:de:9c yes 0.00
1 1 98:09:cf:75:bb:97 no 24.58
1 1 9a:a7:9b:c8:f8:b8 no 243.68
1 5 a2:21:f8:e9:27:2e no 0.66
1 2 aa:46:d9:c3:80:aa no 0.22
1 1 b4:fb:e4:2c:fd:82 no 33.96
1 1 b4:fb:e4:2c:fd:82 no 51.88
1 1 b4:fb:e4:2c:fd:83 no 18.22
2 1 b4:fb:e4:b7:e5:09 no 104.30
1 1 b4:fb:e4:b7:e5:09 no 73.60
1 1 b4:fb:e4:b7:e5:09 no 26.52
1 1 b8:27:eb:d7:c0:aa no 259.61
1 1 bc:33:29:28:bd:2a no 25.28
1 1 bc:d0:74:0d:d8:00 no 3.31
2 5 be:d3:72:4c:24:9f yes 0.00
2 2 c2:cc:9e:1a:fe:68 yes 0.00
1 1 c4:29:96:06:9f:2c no 50.15
1 1 cc:98:8b:39:1f:64 no 4.30
1 1 cc:98:8b:c5:53:f1 no 29.42
1 1 d4:f5:47:85:fd:38 no 29.23
1 1 d6:ee:4e:57:c6:96 no 5.53
1 1 dc:2c:6e:31:28:d7 no 149.65
1 1 dc:2c:6e:31:28:d7 no 25.01
1 1 dc:2c:6e:31:28:d7 no 0.00
1 1 dc:2c:6e:31:28:d9 no 17.91
1 1 dc:2c:6e:31:28:da no 17.91
1 1 dc:2c:6e:31:28:db no 17.91
1 1 dc:2c:6e:31:28:dc no 17.91
4095 1 f0:1f:af:d4:21:ab yes 0.00Note the last line - 4095 entries for the local interface.
Very similar output is from node 2 and node 3 - just with different mac address.
/etc/network/interfaces (node 1):
Code:
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!
auto lo
iface lo inet loopback
auto eno1
iface eno1 inet manual
auto eno2
iface eno2 inet manual
auto bond0
iface bond0 inet manual
bond-slaves eno1 eno2
bond-miimon 100
bond-mode 802.3ad
bond-xmit-hash-policy layer2+3
bond-miimon 100
bond-downdelay 200
bond-updelay 200
bond-lacp-rate 1
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
auto vmbr0.16
iface vmbr0.16 inet static
address 192.168.16.1/24
gateway 192.168.16.254
auto vmbr0.17
iface vmbr0.17 inet static
address 192.168.17.1/24Only node1 have eth bond configured - other nodes have single NICs as part of the vmbr0 bridge without any bonding.
ip l (node 1):
Code:
root@wowza:~# ip l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:d4:21:ab brd ff:ff:ff:ff:ff:ff
altname enp2s0f0
3: eno2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:d4:21:ab brd ff:ff:ff:ff:ff:ff permaddr f0:1f:af:d4:21:ac
altname enp2s0f1
5: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:d4:21:ab brd ff:ff:ff:ff:ff:ff
6: vmbr0.16@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:d4:21:ab brd ff:ff:ff:ff:ff:ff
7: vmbr0.17@vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:d4:21:ab brd ff:ff:ff:ff:ff:ff
8: tap1008i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr1008i0 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 4e:cd:44:27:ba:b3 brd ff:ff:ff:ff:ff:ff
9: fwbr1008i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 5a:a6:b0:b7:57:c7 brd ff:ff:ff:ff:ff:ff
10: fwpr1008p0@fwln1008i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether c2:cc:9e:1a:fe:68 brd ff:ff:ff:ff:ff:ff
11: fwln1008i0@fwpr1008p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr1008i0 state UP mode DEFAULT group default qlen 1000
link/ether ea:5d:3a:aa:01:68 brd ff:ff:ff:ff:ff:ff
12: tap1008i1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr1008i1 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether aa:fc:53:fa:25:33 brd ff:ff:ff:ff:ff:ff
16: tap1004i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr1004i0 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 22:11:b9:c6:a6:84 brd ff:ff:ff:ff:ff:ff
20: tap1009i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master fwbr1009i0 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 3a:cf:53:4f:ed:c4 brd ff:ff:ff:ff:ff:ff
45: fwbr1009i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether aa:2a:a7:36:87:04 brd ff:ff:ff:ff:ff:ff
46: fwpr1009p0@fwln1009i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether be:d3:72:4c:24:9f brd ff:ff:ff:ff:ff:ff
47: fwln1009i0@fwpr1009p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr1009i0 state UP mode DEFAULT group default qlen 1000
link/ether f6:33:a3:2f:b9:ef brd ff:ff:ff:ff:ff:ff
48: fwbr1008i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 46:d8:d2:ed:d8:4f brd ff:ff:ff:ff:ff:ff
49: fwpr1008p1@fwln1008i1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 7e:34:36:02:4a:58 brd ff:ff:ff:ff:ff:ff
50: fwln1008i1@fwpr1008p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr1008i1 state UP mode DEFAULT group default qlen 1000
link/ether fe:f0:7b:a0:ae:26 brd ff:ff:ff:ff:ff:ff
51: fwbr1004i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 02:84:34:bb:d5:a7 brd ff:ff:ff:ff:ff:ff
52: fwpr1004p0@fwln1004i0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 96:5b:45:89:de:9c brd ff:ff:ff:ff:ff:ff
53: fwln1004i0@fwpr1004p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master fwbr1004i0 state UP mode DEFAULT group default qlen 1000
link/ether 1e:c0:9b:4c:3e:f9 brd ff:ff:ff:ff:ff:ff
57: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether f0:1f:af:d4:21:ab brd ff:ff:ff:ff:ff:ffNode 1 is connected using two ethernet cables to Unifi USW-24-G1 (LACP bond).
Node 2 and 3 are connected to MikroTik RB5009UG which is then connected to the USW-24-G1.
All nodes are running up-to-date proxmox 7.2-11 (no subscription) with kernel 5.15.60-1-pve.
Is this a normal behavior or am I doing something wrong?
Any feedback is appreciated!