option to disable MAC learning on bridges

[systemctl]

Hello,

https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_7.3 - Add option to disable MAC learning on bridges (the guest-mac addresses are added statically, no broadcast packets are flooded to those ports, thus no spurious answers are replied, which broke certain hoster network setups)

Could you please tell me where exactly this option was added? I'm looking for it in version 7.4. Is it not available in the GUI ( at least i can't find it ). Can it only be disabled via CLI?

~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.104-1-pve)
pve-manager: 7.4-3 (running version: 7.4-3/9002ab8a)
pve-kernel-5.15: 7.4-1
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.104-1-pve: 5.15.104-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.35-3-pve: 5.15.35-6
pve-kernel-5.13.19-6-pve: 5.13.19-15
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-2
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-4
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-1
libpve-rs-perl: 0.7.5
libpve-storage-perl: 7.4-2
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.1-1
proxmox-backup-file-restore: 2.4.1-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.1-1
proxmox-widget-toolkit: 3.6.5
pve-cluster: 7.3-3
pve-container: 4.4-3
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-1
pve-firmware: 3.6-4
pve-ha-manager: 3.6.0
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-1
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.9-pve1

 

[mira]

This option can only be set directly in /etc/network/interfaces.
Add a line containing bridge-disable-mac-learning 1 to your bridge. Then reload it either via the GUI or the command line with ifreload -a.

 

[systemctl]

Ty for your reply.
However, I'm having something to clarify. I'm currently studying patches on this topic, for example, here https://lists.proxmox.com/pipermail/pve-devel/2021-September/050089.html, they talk about automatic management of the FDB table in case MAC learning is disabled. I also see code related to disabling learning and flood on bridge ports. I'm not strong in programming, so could you explain how this works in conjunction with disabling "bridge-disable-mac-learning" in interfaces? Does PVE parse the "bridge-disable-mac-learning" option from interfaces config, if present, disable learning and flood on all ports, and add static entries to the FDB table? Or am I missing something?


PS: based on the tests I made, it appears to be the case, but I still await your confirmation. By the way, "ifreload -a" didn't help, I had to reboot the entire node to see any changes in /sys/class/net/tapN/brport/learning / unicast_flood

 

[mira]

Does PVE parse the "bridge-disable-mac-learning" option from interfaces config, if present, disable learning and flood on all ports, and add static entries to the FDB table? Or am I missing something?

Yes, exactly. When that option is parsed and set to `1` both learning and flood are disabled.
And whenever a guest is started, its MAC address is added to the FDB and removed when the guest is stopped.

Pretty sure already running guests are not affected, but only newly started ones. But I'd have to check the source to verify this.

 

[systemctl]

Yes, exactly. When that option is parsed and set to `1` both learning and flood are disabled.
And whenever a guest is started, its MAC address is added to the FDB and removed when the guest is stopped.

Pretty sure already running guests are not affected, but only newly started ones. But I'd have to check the source to verify this.

Indeed, stop-start works fine after ifreload -a.

But i have a question..after disabling learning and flood i still can see unicast flood for example ( 00:25:90:47:27:b4 vmbr0 mac address )

# tcpdump -en -i vmbr0 -Q in 'not (broadcast or multicast or ether dst 00:25:90:47:27:b4)'
14:19:54.869932 d8:53:9a:15:71:f3 > ac:1f:6b:da:18:4e, ethertype IPv4 (0x0800), length 139: 45.125.65.31.62785 > 195.123.231.38.55555: Flags [P.], seq 382:467, ack 1244, win 1021, length 85
14:19:54.887591 d8:53:9a:15:71:f3 > 0c:c4:7a:9b:73:7c, ethertype IPv4 (0x0800), length 139: 45.125.65.31.62854 > 195.123.231.103.55555: Flags [P.], seq 382:467, ack 1244, win 1021, length 85
14:19:54.921185 d8:53:9a:15:71:f3 > ac:1f:6b:da:18:4e, ethertype IPv4 (0x0800), length 683: 45.125.65.31.62785 > 195.123.231.38.55555: Flags [P.], seq 467:1096, ack 1505, win 1026, length 629
14:19:54.938433 d8:53:9a:15:71:f3 > 0c:c4:7a:9b:73:7c, ethertype IPv4 (0x0800), length 683: 45.125.65.31.62854 > 195.123.231.103.55555: Flags [P.], seq 467:1096, ack 1505, win 1026, length 629

But I think I'm not checking it correctly because I'm doing it on the actual bridge, and unicast flood will continue to come to it anyway. Should perhaps I check it on the lower-level interfaces (like tap interfaces) instead?

 

[mira]

Did you shutdown/start each of the VMs on that bridge?

Are those MAC addresses coming from one of the VMs? The flood/learning is only disabled for guests on that bridge since those MAC addresses are known.
What kind of packets were those?
Where did they come from and was the destination somewhere on the bridge?

 

[spirit]

Indeed, stop-start works fine after ifreload -a.

But i have a question..after disabling learning and flood i still can see unicast flood for example ( 00:25:90:47:27:b4 vmbr0 mac address )

# tcpdump -en -i vmbr0 -Q in 'not (broadcast or multicast or ether dst 00:25:90:47:27:b4)'
14:19:54.869932 d8:53:9a:15:71:f3 > ac:1f:6b:da:18:4e, ethertype IPv4 (0x0800), length 139: 45.125.65.31.62785 > 195.123.231.38.55555: Flags [P.], seq 382:467, ack 1244, win 1021, length 85
14:19:54.887591 d8:53:9a:15:71:f3 > 0c:c4:7a:9b:73:7c, ethertype IPv4 (0x0800), length 139: 45.125.65.31.62854 > 195.123.231.103.55555: Flags [P.], seq 382:467, ack 1244, win 1021, length 85
14:19:54.921185 d8:53:9a:15:71:f3 > ac:1f:6b:da:18:4e, ethertype IPv4 (0x0800), length 683: 45.125.65.31.62785 > 195.123.231.38.55555: Flags [P.], seq 467:1096, ack 1505, win 1026, length 629
14:19:54.938433 d8:53:9a:15:71:f3 > 0c:c4:7a:9b:73:7c, ethertype IPv4 (0x0800), length 683: 45.125.65.31.62854 > 195.123.231.103.55555: Flags [P.], seq 467:1096, ack 1505, win 1026, length 629

But I think I'm not checking it correctly because I'm doing it on the actual bridge, and unicast flood will continue to come to it anyway. Should perhaps I check it on the lower-level interfaces (like tap interfaces) instead?

but I think that it shouldn't be forwarded to the vm ? (can you verify with a tcpdump in the vm).

AFAIK, to fully disable the flood enter through the physical interfaces, all vms on the bridge need to be restarted (all vm bridge ports need to have flood disabled), and vlan-aware need be enable on the bridge too. (this is a limitation of linux kernel)

if this 2 conditions are ok, the physical shoudln't be in prosmicous mode. (I think it should be logged)

reference:
https://git.backbone.ws/kolan/backbone-sources/commit/2796d0c648c9

 

[systemctl]

Did you shutdown/start each of the VMs on that bridge?

Are those MAC addresses coming from one of the VMs? The flood/learning is only disabled for guests on that bridge since those MAC addresses are known.
What kind of packets were those?
Where did they come from and was the destination somewhere on the bridge?

Yes, all vm are restarted ( to be honest i've restarted whole node )
There is only 1 working test VM on that node, to make it easier to debug

And I'm a bit confused, to be honest. Let's start with how to properly check for the absence of unicast flood. On which interface should I monitor the packets? It's definitely not inside the VM because the firewall won't let the packets through. But as far as I understand, monitoring on vmbr0 is not entirely correct either because unicast flood will still come to it. Disabling unicast_flood and learning should remove unwanted traffic between bridge ports. Therefore, am I correct to assume that I should look at tapNi0? For example, using tcpdump -en -i tap129i0 -Q in 'not (broadcast or multicast)'. are my thoughts incorrect?

and vlan-aware need be enable on the bridge too. (this is a limitation of linux kernel)

is it necessary? tbh never heard of it, we don't use vlan..
And i'll check this topic a little deeper, thank you

 

[spirit]

Yes, all vm are restarted ( to be honest i've restarted whole node )
There is only 1 working test VM on that node, to make it easier to debug

And I'm a bit confused, to be honest. Let's start with how to properly check for the absence of unicast flood. On which interface should I monitor the packets? It's definitely not inside the VM because the firewall won't let the packets through. But as far as I understand, monitoring on vmbr0 is not entirely correct either because unicast flood will still come to it. Disabling unicast_flood and learning should remove unwanted traffic between bridge ports. Therefore, am I correct to assume that I should look at tapNi0? For example, using tcpdump -en -i tap129i0 -Q in 'not (broadcast or multicast)'. are my thoughts incorrect?

yes, tcpdump the tap interface should be enough.

is it necessary? tbh never heard of it, we don't use vlan..
And i'll check this topic a little deeper, thank you

yep. (if you don't want to receive flood traffic from outside to your vmbr.

from the commit:
"
Since the above functionality depends on full static configuration,
we have also require that vlan filtering be enabled to take
advantage of this. The reason is that the bridge has to be
able to receive and process VLAN-tagged frames and the there
are only 2 ways to accomplish this right now: promiscuous mode
or vlan filtering."

Note that it should be fine, even if you don't use any vlan. (If you are able to test, it could be great

 

[systemctl]

yep. (if you don't want to receive flood traffic from outside to your vmbr.

Okay, i did it

bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
bridge-disable-mac-learning 1

but i don't see any changes ( checking brctl show )
ip -details link shows "promiscuity 0" for vmbr0. Same values is on another node with flooding and learning turned on. So i can't see any difference here.

At first i thought that 5: tap129i0: <BROADCAST,MULTICAST,[B]PROMISC[/B],UP,LOWER_UP> is the things, but i have the same PROMISC option on tap interfaces on another node, that has no learning and flood disabled at all.

 

[spirit]

Okay, i did it

bridge-ports eno1
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
bridge-disable-mac-learning 1

but i don't see any changes ( checking brctl show )

do you have restart/reboot ? (reload is not working fine with changing from non-vlan-aware / vlan aware)

you can do a "bridge -c vlan show" to see the diff.

ip -details link shows "promiscuity 0" for vmbr0. Same values is on another node with flooding and learning turned on. So i can't see any difference here.

At first i thought that 5: tap129i0: <BROADCAST,MULTICAST,[B]PROMISC[/B],UP,LOWER_UP> is the things, but i have the same PROMISC option on tap interfaces on another node, that has no learning and flood disabled at all.

the promisc should be disabled on physical interfacee.

for tap, proxmox set them as promisc in any case (don't think it's a problem with mac leaning disable && unicast_flood off on tap).
/usr/share/perl5/PVE/Network.pm
run_command(['/sbin/ip', 'link', 'set', $iface, 'up', 'promisc', 'on', 'mtu', $bridgemtu]);

do you see any anycast flood with tcpdump on tap interfaces ?

 

[systemctl]

do you have restart/reboot ? (reload is not working fine with changing from non-vlan-aware / vlan aware)

you can do a "bridge -c vlan show" to see the diff.

the promisc should be disabled on physical interfacee.

for tap, proxmox set them as promisc in any case (don't think it's a problem with mac leaning disable && unicast_flood off on tap).
/usr/share/perl5/PVE/Network.pm
run_command(['/sbin/ip', 'link', 'set', $iface, 'up', 'promisc', 'on', 'mtu', $bridgemtu]);

do you see any anycast flood with tcpdump on tap interfaces ?

It seems like I'm starting to understand, thank you. The output of the bridge -c vlan show command is
# bridge -c vlan show

port              vlan-id 
eno1              1 PVID Egress Untagged
                  2-4094
vmbr0             1 PVID Egress Untagged
tap129i0          1 PVID Egress Untagged
fwbr129i0         1 PVID Egress Untagged
fwpr129p0         1 PVID Egress Untagged
                  2-4094
fwln129i0         1 PVID Egress Untagged

The output of ip -details link on the node with bridge-vlan-aware yes and bridge-disable-mac-learning 1 is

3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:47:27:b4 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9216
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:47:27:b4 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535

As we can see, the promiscuous mode has been disabled, which is great. On the node where nothing was configured, we have the following result -

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:15:a7:22 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:15:a7:22 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535

indicating that the physical port is in promiscuous mode.

The main question now is, how dependent is the disabling of promiscuous mode on the physical port and the disabling of bridge learning and unicast_flood on the Proxmox ports? As I understand it, the second action, without the first one, won't prevent the flood from reaching beyond the bridge, as the physical port will still allow this flood to pass through. With promiscuous mode enabled, the flood will be restricted at the physical device. Is this correct?

And if that's the case, why don't we simply activate VLAN filtering to put the physical device in non-promiscuous mode, which would cut off all flood traffic even before reaching the bridge? Because according to the information provided in https://git.backbone.ws/kolan/backbone-sources/commit/2796d0c648c9, "if the user places the bridge device in promiscuous mode, all ports are placed in promiscuous mode regardless of the changes to flooding and learning."
I apologize for any potentially silly questions; I just want to understand how it works.

 

[spirit]

It seems like I'm starting to understand, thank you. The output of the bridge -c vlan show command is
# bridge -c vlan show

port              vlan-id
eno1              1 PVID Egress Untagged
                  2-4094
vmbr0             1 PVID Egress Untagged
tap129i0          1 PVID Egress Untagged
fwbr129i0         1 PVID Egress Untagged
fwpr129p0         1 PVID Egress Untagged
                  2-4094
fwln129i0         1 PVID Egress Untagged

The output of ip -details link on the node with bridge-vlan-aware yes and bridge-disable-mac-learning 1 is

3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:47:27:b4 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9216
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 00:25:90:47:27:b4 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535

As we can see, the promiscuous mode has been disabled, which is great. On the node where nothing was configured, we have the following result -

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:15:a7:22 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 9216
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether ac:1f:6b:15:a7:22 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535

indicating that the physical port is in promiscuous mode.

ok great !

The main question now is, how dependent is the disabling of promiscuous mode on the physical port and the disabling of bridge learning and unicast_flood on the Proxmox ports? As I understand it, the second action, without the first one, won't prevent the flood from reaching beyond the bridge, as the physical port will still allow this flood to pass through. With promiscuous mode enabled, the flood will be restricted at the physical device. Is this correct?

if option bridge-disable-mac-learning , is disable learning (+manually register mac address in bridge fdb) + disable unicast_flood on each vm port. (but not physical port , because we need to learn outside mac)

This is enough to block traffic at bridge level. (tcpdump -i tap , it shouldn't see unknown destination traffic)

And if that's the case, why don't we simply activate VLAN filtering to put the physical device in non-promiscuous mode, which would cut off all flood traffic even before reaching the bridge? Because according to the information provided in https://git.backbone.ws/kolan/backbone-sources/commit/2796d0c648c9, "if the user places the bridge device in promiscuous mode, all ports are placed in promiscuous mode regardless of the changes to flooding and learning."
I apologize for any potentially silly questions; I just want to understand how it works.

the 2 conditions to disable prosmisc are:
- you need vlan-aware enabled on bridge.
- you need to have maximum 1 interface with flooding/learning enabled on the bridge. (the physical interface)

 

[systemctl]

ok great !


if option bridge-disable-mac-learning , is disable learning (+manually register mac address in bridge fdb) + disable unicast_flood on each vm port. (but not physical port , because we need to learn outside mac)

This is enough to block traffic at bridge level. (tcpdump -i tap , it shouldn't see unknown destination traffic)


the 2 conditions to disable prosmisc are:
- you need vlan-aware enabled on bridge.
- you need to have maximum 1 interface with flooding/learning enabled on the bridge. (the physical interface)

Thank you a lot for guiding me.
To summarize, taking into account the requirement "you need to have maximum 1 interface with flooding/learning enabled on the bridge (the physical interface)," in order to enable promiscuous mode on the physical port, you still need to disable MAC learning and unicast_flood on all bridge ports (except the physical one), is that correct?

 

[spirit]

Thank you a lot for guiding me.
To summarize, taking into account the requirement "you need to have maximum 1 interface with flooding/learning enabled on the bridge (the physical interface)," in order to enable promiscuous mode on the physical port, you still need to disable MAC learning and unicast_flood on all bridge ports (except the physical one), is that correct?

yes, exactly.

 

[itNGO]

Yes, exactly. When that option is parsed and set to `1` both learning and flood are disabled.
And whenever a guest is started, its MAC address is added to the FDB and removed when the guest is stopped.

Pretty sure already running guests are not affected, but only newly started ones. But I'd have to check the source to verify this.

Sorry to bring this up again. Migrating the guests between Nodes would also trigger the change?

bridge-disable-mac-learning 1

 

[itNGO]

Sorry to bring this up again. Migrating the guests between Nodes would also trigger the change?

bridge-disable-mac-learning 1

Answering my own question... yes migrate between nodes is enough.
Oh and you loose any Mac-Adress-Spoofing possibility. Like using opnsense in HA-Mode with VIPs.... cause only the known Mac-Adresses in are allowed....