Proxmox firewall showing drop logs for another VM

[Blackworm]

Hello everyone,

I searched the forum but couldn't find a similar post...

I have started to set up firewalls on my VMs. Simply, I found myself seeing some very strange logs.

To explain, I have 3 VMs

HostA = a simple router. ip = 10.0.0.1
HostB = a server hosting a web service on 443 port. ip = 10.0.0.10
HostC = another server. ip = 10.0.0.20

I've enabled the firewall only on HostC for now, with INPUT POLICY DROP and a rule to allow SSH. VMs HostA and HostB do not have the firewall enabled.

However, on the HostC firewall logs I get this :

policy DROP: IN=fwbr2020i0 OUT=fwbr2020i0 PHYSIN=fwln2020i0 PHYSOUT=tap2020i0 MAC=aa:bb:cc:dd:ee:ff:ab:bc:cd:de:ef:fg:08:00 SRC=10.0.0.1 DST=10.0.0.10 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=15878 DF PROTO=TCP SPT=47156 DPT=443 SEQ=3062970246 ACK=0 WINDOW=32120 SYN

I checked for duplicate MACs and found that all my network cards have a unique MAC address. For your information, I'm running Virtual Environment 8.2.2. The strange thing is that I do have access to the 10.0.0.10:443 web server.

It's as if the packets were misrouted from time to time


Thanks to those who will take the time to read me

 

[ocroquette]

I have a very similar problem. A VM gets packets that it is not supposed to receive. The destination IP and MAC are those of another VM. The firewall of the "wrong" VM blocks and logs them. This is a low traffic environment, and it happens around every hour. Since this connection is typically a user trying to open an web page, it is very annoying. The workaround is to reload the page a few times. Then at some point the packets are sent to the right VM again. I checked:

• the MAC addresses of the VMs, no duplicates
• the ARP caches, nothing suspiscious
• the ARP requests and replies, nothing suspiscious

It looks like the issue is in the virtual switch.

@Blackworm Did you find the issue back then?

 

[ocroquette]

Related links:
https://forum.proxmox.com/threads/option-to-disable-mac-learning-on-bridges.129318/
https://forum.proxmox.com/threads/dynamic-forwarding-database-on-network-changes.144489/

I will try to set "bridge-disable-mac-learning" to "1" during the next downtime of this host.

 

[ocroquette]

Update: today, the whole communication of this virtual switch broke down. Pinging from the host to the VM and between the VM failed (no route to host). So I had my downtime I took the opportunity to set bridge-disable-mac-learning to 1, and restarted the host. After the restart, the communication was working again.
I will keep an eye on the log files to see if there are still misrouted packets.
Everythink points to a misbehaviour of the virtual bridge. It could be due to misconfiguration, but I don't think so, since it is pretty vanilla, nothing fancy, and the fact that it broke down. It looks more like a bug that can cause sporadic problems, small or big.

 

[shanreich]

If the MAC of the network interface isn't in the fdb of the bridge, then packets for that interface will get flooded to all bridge ports. This exact behavior is what you're disabling with bridge-disable-mac-learning. So, as long as this setting is enabled, seeing those packets pop up sporadically is expected behavior.

Do you have INPUT policy REJECT set by chance?

 

[ocroquette]

Yes, I have INPUT policy set as REJECT with logging.

 

[ocroquette]

Now I understand what flooding means... I thought it was some kind of broadcast, like ARP requests, but "real" packets are sent. So yes, the log entries do make sense.
What I still don't understand is why this mechanism stopped working sporadically.

 

[shanreich]

Yes, I have INPUT policy set as REJECT with logging.

What's possible then is that when the packet gets flooded to all ports, the firewall sends a REJECT (= TCP RST) for your TCP packet and then the TCP connection gets closed. It's a race condition depending on whether the TCP ACK from the web server first arrives at your client or the TCP RST from the firewall.

 

[Blackworm]

TBH, I did found any solution. As you say, the VM gets some packets that should not be there.
I don't like my solution but for now I disabled my router firewall...

After reading the discussion, I finally didn't understand: should bridge-disable-mac-learning be enabled or disabled?

Also, I quite agree with @ocroquette that it could be a bug. My configuration is quite trivial, I have a few subnetworks but all ips are well defined in a lxc dns

 

[ocroquette]

What's possible then is that when the packet gets flooded to all ports, the firewall sends a REJECT (= TCP RST) for your TCP packet and then the TCP connection gets closed. It's a race condition depending on whether the TCP ACK from the web server first arrives at your client or the TCP RST from the firewall.

Of course. It would also explain why the issue occurs more often lately. I have more and more VMs connected to this switch. So the actual VM has to compete with always more firewalls. Also, the load on the host grows, causing more jitter.

 

[ocroquette]

TBH, I did found any solution. As you say, the VM gets some packets that should not be there.
I don't like my solution but for now I disabled my router firewall...

After reading the discussion, I finally didn't understand: should bridge-disable-mac-learning be enabled or disabled?

Also, I quite agree with @ocroquette that it could be a bug. My configuration is quite trivial, I have a few subnetworks but all ips are well defined in a lxc dns

The unwanted packets are part of a "pragmatic" discovery mechanism, called MAC-learning. You can disable it by setting bridge-disable-mac-learning to 1. Then the MAC lists will be maintained statically based on the interfaces connected to the bridge, instead of dynamically by sending packets to machines that do not expect them. At least this is what I understood.

 

[shanreich]

After reading the discussion, I finally didn't understand: should bridge-disable-mac-learning be enabled or disabled?

Depends, if the interfaces on that bridge only use the MAC address from the VM config, then you should be good to go. If you had, for instance, nested PVE with a bridge - then you would need MAC learning since otherwise the bridge would not learn the additional MAC addresses from the guests of the nested PVE instance. Generally I'd recommend turning MAC learning off unless you have a reason to turn it on, since otherwise any guest can send packets with spoofed MAC addresses that the bridge then learns.

If you disable it, then you should not see any unwanted packets since there will be no flooding for unicast(!) packets. It's also how physical switches function, so it's not really a bug. With disable MAC learning we manually create entries in the bridge fdb and tell the bridge to only use those static mappings.

Of course. It would also explain why the issue occurs more often lately. I have more and more VMs connected to this switch. So the actual VM has to compete with always more firewalls. Also, the load on the host grows, causing more jitter.

You could try setting the default INPUT policy of all VMs to DROP, then the problem should not occur since they silently discard the packets instead of replying with a TCP RST.

 

[ocroquette]

Generally I'd recommend turning MAC learning off unless you have a reason to turn it on, since otherwise any guest can send packets with spoofed MAC addresses that the bridge then learns.

That sounds good, I will follow that advice. Quick questions:

• If I set "bridge-disable-mac-learning" to 1 in /etc/network/interfaces, what is the best way to let Proxmox apply this new setting, if possible without downtime or (noticeable) interruption for the users?
• How can I verify the current effective value of bridge-disable-mac-learning for a specific bridge?

You could try setting the default INPUT policy of all VMs to DROP, then the problem should not occur since they silently discard the packets instead of replying with a TCP RST.

I want to avoid that, for the reasons you mentioned above, but also because it makes troubleshooting more cumbersome. For instance, if you have a typo in a port number, and the policy is DROP, it adds to the confusion, since it can also indicate general networking or routing issues.