[SOLVED] Somebody help me make sense of PVE firewall, getting DROPs for traffic that shouldn't be on the host.

[lucius_the]

Hi all,
Running PVE 7.4.16. Have (quite some) experience with Mikrotik ROS firewalls (which is much like iptables just with a nice GUI - only stating this to let you know I'm not a complete newbie).

With PVE firewall, though I have some things I can't wrap my head around.

I've turned on the firewall (on cluster level, then on the node itself, then on 2 CT-s). Firewall is really a default setup, with policy DROP in input and ACCEPT in output.
I've turned on logging on input to see if anything else it trying to access something on those CTs that I haven't allowed.
So now I get things like this in the log:

100 6 veth100i0-IN 09/Sep/2023:20:00:27 +0200 policy DROP: IN=fwbr100i0 OUT=fwbr100i0 PHYSIN=fwln100i0 PHYSOUT=veth100i0 MAC=72:64:58:8c:99:56:8e:84:98:00:9d:78:08:00 SRC=10.10.40.135 DST=10.10.40.145 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=39627 DF PROTO=TCP SPT=62978 DPT=6162 SEQ=3798276170 ACK=0 WINDOW=62720 SYN
114 6 veth114i0-IN 09/Sep/2023:20:00:27 +0200 policy DROP: IN=fwbr114i0 OUT=fwbr114i0 PHYSIN=fwln114i0 PHYSOUT=veth114i0 MAC=72:64:58:8c:99:56:8e:84:98:00:9d:78:08:00 SRC=10.10.40.135 DST=10.10.40.145 LEN=52 TOS=0x00 PREC=0x00 TTL=128 ID=39627 DF PROTO=TCP SPT=62978 DPT=6162 SEQ=3798276170 ACK=0 WINDOW=62720 SYN

All fine, except these IP addresses have nothing to do with those CT-s. I have a bunch of other examples in the logs. These other examples have addresses that have nothing to do with either these CT-s or any other IP address on PVE this host. Which I find quite confusing.

So... what's going on in here ?
Why and how is PVE firewall on my CT-s even seeing traffic that nothing to do with those CT-s. Has anyone else seen things like this happening ?

This is my network setup:
auto bond0
iface bond0 inet manual
bond-slaves eno12399np0 eno8303
bond-miimon 100
bond-mode active-backup
bond-primary eno12399np0
mtu 9000

auto vmbr0
iface vmbr0 inet static
address 10.10.40.41/23
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4094
mtu 9000

 

[lucius_the]

Figured it out soon enough, coming here to answer my own question in hope it helps someone.

Turns out I had to get back to basics and understand how L2 works
What happens is that switches learn MAC addresses by listening to ARP traffic, or simply by monitoring on which ports a certain MAC appears as the source (sending) MAC. The switch then checks if it knows about that MAC and if not it adds it to the table with the info on which port that MAC address lives. Now, switches have limited RAM and their MAC tables are limited in size. So, switches have a timeout for how long that MAC will remain in its table - with a fairly low default, that is measured in seconds, not hours. On some switches you can up that, but on my HP Aruba switch the default is 4 minutes and I can set a maximum to 6 minutes and that's it.

What happens is I have PC-s in my network that are asleep. They wake up on scheduled time to start a backup. So, hours may pass before that device had any reason to be active on the network. When the PC resumes from sleep and backup starts, it tries to connect to network backup storage and that's a TCP SYN packet. On L2 that means some packet is sent from source MAC to destination MAC, but that destination MAC hasn't been transmitting on the network for a long time.

So what happens if the switch doesn't already know where the destination MAC is located, what will it do ? It will flood the packet on all ports. But how can a network device send a packet to some IP without sending an ARP packet to find out what the destination MAC is, in the first place ? Well, turns out that operating systems have a much larger timeout for it's ARP table (a table that resolves IP addresses to MAC addresses) than a switch does for its MAC table. So the PC already knows the destination MAC and doesn't send any ARP request to find out what the destination MAC is, which would help the switch to also learn where that MAC lives. The switch (with it's default MAC table timeout of 4 minutes, in my case at least) doesn't know where the destination MAC is and sends the packet on all ports in an effort to find where to send the rest of the packets. As the destination network device responds, the switch then learns the MAC and stops flooding traffic to all ports.

That's why I'm seeing what seems like something terribly wrong. I'm seeing a firewall DROP for a TCP SYN from net device 1 to net device 2 where both net devices have nothing to do with any network device hosted on Proxmox. The switch didn't know where to send the packet so it ended up everywhere, including the port Proxmox host is connected to. So Proxmox gets that TCP SYN packet, the first packet in conversation. And that's the only packet I'm seeing in the logs. As soon as the target responds and the switch learns it's MAC address, no further traffic is being flooded on all ports and it's no longer reaching the Proxmox host. That's why I'm seeing only the first packet.

But there's another quirk here. Proxmox host still shouldn't be seeing all packets. Network drivers normally drop traffic not destined to itself (traffic not destined to our MAC address), so the OS never even gets to see that traffic. It can basically be dropped in the ethernet chip itself, so that such traffic doesn't waste any CPU time for such packets on the host that has nothing to do with that traffic. So how does this happen ? For a host to be able to receive network traffic not destined to itself it must be in a so-called "promiscuous" mode of operation. Turns out my network device in Proxmox is in fact in promiscuous mode. Reason for that ? I don't know, but probably because it doesn't host a single MAC address (like a standard network device does) because it's a VM/CT host. And each VM/CT has a MAC address, so Proxmox needs to be able to listen to traffic for more than just one MAC. Maybe network chips can't do more than 1 MAC filtering, or maybe not all of them can, or something like that - I don't really know. I am curios though, why Proxmox doesn't still filter out traffic not destined to any of the MAC addresses that it hosts, because it could do that in software. Maybe it's is because I'm using a bond, I don't know -> I haven't done any further digging around that because having a few packets dropped isn't a big deal. And I should segment my network in more VLANs anyway.

This was enough for me to understand why I'm seeing the firewall drops I'm seeing.

This is actually very basic networking, but it did take take me some time to remember how things work on L2 and + realising that promisc mode is on, etc, to finally understanding what's going on. And I'm sharing this here for potential other people interested in understanding the same thing.

This can, in theory, be resolved by means of:
- increasing MAC table timeout on the switch (if the switch supports it) to a value that is higher than ARP cache timeout on the clients in this ethernet segment
- using static MAC, or static MAC learning (if the switch and the environment permits that)
But, in my case it's not worth the trouble.

 

[lucius_the]

To be honest, I still wonder why that TCP SYN packet ends up in firewall for all CT-s. It should, at least, be filtered in the main (parent/host) firewall, not the one for a certain CT, they really should have no reason to have it's network device in promisc mode, at least not by default...