Bridge fdb (Forward DB) fills with 4095 mac for each VM (more than 32000 mac per host).

[francoisd]

Hi,

I had a very strange behaviour, that basically completely broke my bridging in Proxmox. (I ended up with kind of 2 isolated set of hosts that were able to exchange ARP info, but not between the 2 sets).

My config: 3 Proxmox hosts in cluster, with on each host:

• 1 Ethernet NIC - Default vlan untagged, and 4 tagged vlans
• 1 vlan aware bridge (vmbr0) containing that network interface

During my problem, I noticed that the "bridge fdb" was containing more than 32800 entries.

After a full reboot of the hosts (it's a cluster of 3, but quite irrelevant here since I had the problem even between VM/containers on the same host, regardless of VM or Container), I got my ARP working properly again, but I still see those fdb tables filling with 4094 unknown MAC addresses for each VM or Container :

root@hystou1:~# bridge fdb show | wc -l
20666
root@hystou1:~# bridge fdb show | uniq -w 30 -c | sort -n | tail
      2 e2:cf:74:ae:b9:c3 dev fwln211i0 vlan 1 master fwbr211i0 permanent
      2 fe:0c:89:67:50:fc dev fwln114i0 vlan 1 master fwbr114i0 permanent
      2 fe:6b:cf:1d:bb:09 dev veth211i0 vlan 1 master fwbr211i0 permanent
      2 fe:ac:1e:ad:78:ec dev veth101i0 vlan 1 master fwbr101i0 permanent
      2 fe:af:3b:90:8f:c1 dev veth105i0 vlan 1 master fwbr105i0 permanent
   4094 00:e0:4c:68:69:bb dev enp1s0 vlan 4094 master vmbr0 permanent
   4095 0e:af:56:62:62:95 dev fwpr105p0 vlan 4094 master vmbr0 permanent
   4095 36:6c:76:87:c7:48 dev fwpr114p0 vlan 4094 master vmbr0 permanent
   4095 72:6e:ac:d7:32:35 dev fwpr211p0 vlan 4094 master vmbr0 permanent
   4095 ee:10:2a:50:a6:1d dev fwpr101p0 vlan 4094 master vmbr0 permanent
root@hystou1:~#

root@hystou2:~# bridge fdb show | wc -l
33038
root@hystou2:~# bridge fdb show | uniq -w 30 -c | sort -n | tail
      4 3e:4d:09:1d:c0:dd dev enp1s0 vlan 1 master vmbr0
      5 ce:d8:65:89:6f:a6 dev tap111i0 vlan 129 master vmbr0
   4094 00:e0:4c:68:65:b9 dev enp1s0 vlan 4094 master vmbr0 permanent
   4095 2e:9d:93:16:28:8a dev fwpr107p0 vlan 4094 master vmbr0 permanent
   4095 52:99:0e:e7:89:66 dev fwpr202p0 vlan 4094 master vmbr0 permanent
   4095 6e:be:7b:ea:b5:3e dev fwpr108p0 vlan 4094 master vmbr0 permanent
   4095 72:47:ec:20:9b:f6 dev fwpr115p0 vlan 4094 master vmbr0 permanent
   4095 7a:84:fe:d1:f1:25 dev fwpr102p0 vlan 4094 master vmbr0 permanent
   4095 92:83:41:14:f8:ab dev fwpr112p0 vlan 4094 master vmbr0 permanent
   4095 ca:56:c9:78:68:f8 dev tap111i0 vlan 4094 master vmbr0 permanent
root@hystou2:~#

root@hystou3:~# bridge fdb show | wc -l
28899
root@hystou3:~#  bridge fdb show | uniq -w 30 -c | sort -n | tail
      2 fe:7f:b9:5d:66:51 dev veth203i0 vlan 1 master fwbr203i0 permanent
      4 3e:4d:09:1d:c0:dd dev tap113i0 vlan 1 master vmbr0
      5 ce:d8:65:89:6f:a6 dev enp1s0 vlan 129 master vmbr0
   4094 00:e0:4c:68:01:23 dev enp1s0 vlan 4094 master vmbr0 permanent
   4095 0a:b4:27:ed:58:35 dev fwpr103p0 vlan 4094 master vmbr0 permanent
   4095 42:9a:ba:b5:aa:b4 dev fwpr116p0 vlan 4094 master vmbr0 permanent
   4095 76:51:1d:4e:9b:5c dev tap113i0 vlan 4094 master vmbr0 permanent
   4095 7e:87:30:90:02:dc dev fwpr203p0 vlan 4094 master vmbr0 permanent
   4095 a6:3a:2f:ad:02:b6 dev fwpr117p0 vlan 4094 master vmbr0 permanent
   4095 be:a1:6c:f4:a4:df dev fwpr109p0 vlan 4094 master vmbr0 permanent
root@hystou3:~#

There is only one host that do not trigger those extra 4095 unknown MAC addresses in the fdb, it's a host on which I explicitely untag one of the tagged vlans.

Could someone explain me if this is a healthy behavior, and point me to some possible explanations ?

Thanks !

 

[Alwin]

bridge-vlan-aware yes
bridge-vids 2-4094

When set through the GUI, the VLAN ids are set as well. Maybe that's the difference between the nodes?

 

[francoisd]

And is this normal to have so many thousands of records when checking the bridge fdb with:
bridge fdb show
?

 

[Alwin]

bridge fdb show

I suppose it depends on how many network devices there are. For the command, without any device (dev|br) it will show every entry from every network adapter associated.