[SOLVED] pve-firewall REJECT rule from other VM prevents connection on first try

[elelayan]

Hi,

on a PVE 7.4 cluster, with vmbr0 (vlan aware, stp off), connected to bond0 (2 physical interfaces), with firewall enabled at the cluster level, I have an issue when first opening connections.

a VM protected (100) has a firewall policy in REJECT (so that connection errors are explicit)
a VM client (159) on the same host has no firewall enabled and tries to reach server (a service outside the cluster)

Here is what happens:

1. after some inactivity, when the connection from client to server is attempted the client gets connection refused.
   Observing with tcpdump -a any, I see the SYN packet to open the connection on client's tap159i0, then on bond0, then on physical interface enp25s0f1np1.
   This is all well and good.
2. But I also see the SYN packet duplicated on all other interfaces connected to the bridge.
   In particular it is sent to the protected interface fwpr100p0 then fwln100i0.
   The REJECT rule creates a TCP RST packet by the firewall on the protected interface.
3. The REJECT packet arrives to the client and it severs the connection.
4. Then the SYN,ACK from the actual server arrives, but it's too late, the connection is closed, so the client emits a RST.

What am I doing wrong?

• should I never use input policy REJECT on VMs and always specify the destination address if I want a REJECT rule?
  
• should I disable MAC learning on bridge vmbr0?
• is it some bug in a (combination of) version(s) and upgrading will solve it?

Thanks,

 

[shanreich]

You seem to be running into this bug [1]. The (experimental) proxmox-firewall solves this issue (by not allowing REJECT rules in the input chain of VMs). Dropping this traffic is the only solution currently.

[1] https://bugzilla.proxmox.com/show_bug.cgi?id=4964

 

[elelayan]

Ok, thank you @shanreich for the prompt and sourced reply.

This really got me stumped for a few days...
Is it a corner case or did REJECT not work in general for input policy in VMs?

Is disabling MAC learning on the bridge any help?

Best regards,

 

[elelayan]

The problem seems to have appeared after moving from ifupdown to ifupdown2 and rebooting the host.

 

[shanreich]

The problem seems to have appeared after moving from ifupdown to ifupdown2 and rebooting the host.

That is indeed a bit weird, maybe because the fdb is now recreated and there is no flood of the SYN packet? I'd have to take a closer look, to be honest.

Is it a corner case or did REJECT not work in general for input policy in VMs?

The general issue here is that after routing the packet it might get flooded to other ports. Disabling MAC learning would help in the unicast case, since then there is no unicast flood and therefore no REJECT response from interfaces that have a respective policy.

There are still other cases where this would not work (broadcast, multicast for instance). netfilter actually forbids REJECTING packets at that stage (hence why it is not possible in the new firewall), but due to the way the old firewall works with firewall bridges we can actually reject packets at the INPUT stage, but then bugs like this appear.

 

[elelayan]

Please can somebody add an entry in https://pve.proxmox.com/wiki/Firewall#_tips_and_tricks like

Issues with REJECT rules
Be warned that broad REJECT rules (Input Policy = REJECT or REJECT rule without a specific destination) are dangerous because of packet flood on linux bridge: See https://bugzilla.proxmox.com/show_bug.cgi?id=4964.
Also from a forward compatibility aspect: REJECT rules will not be allowed in proxmox-firewall.
Symptom: connection refused the first time a VM opens a connection, then works for 5 minutes.
Solution: use DROP input policy, DROP rules or REJECT rules with the VMs IP address as destination.