Firewall Rules for Specific Virtual Machines and Linux Containers Affecting VMs with no Firewall

[SillySysAdmin]

I have a Node with about 30 Virtual Machines and 3 Linux Containers. Some VMs have firewalls enabled, and some do not. The firewalls for the enabled machines work as would be intended. However, on occasion, when two Virtual Machines try to establish a connection with each other, a REJECT rule from an unrelated, firewalled VM, will get hit and a non-firewalled machine sending the packet to the other non-firewalled machine will get a Connection Refused error. Normally upon making a second or third attempt, the non-firewalled machine will be able to get to the other non-firewalled machine again without issue.

My firewall log shows firewall rules from firewalled VMs

13 7 tap13i0-IN 10/Dec/2018:21:46:21 -0600 policy REJECT:IN=fwbr13i0 OUT=fwbr130 PHYSIN=fwln13i0 PHYSOUT=tap13i0 MAC=F0:0F:00:69:38:d8:F0:0F:00:42:05:12:08:00 SRC=192.168.32.40 DST=192.168.32.45 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8824 DF PROTO=TCP SPT=51312 DPT=80 SEQ=862487553 ACK=0 WINDOW=29200 SYN 
18 7 tap18i0-IN 10/Dec/2018:21:46:21 -0600 policy REJECT:IN=fwbr18i0 OUT=fwbr18i0 PHYSIN=fwln18i0 PHYSOUT=tap18i0 MAC=F0:0F:00:69:38:d8:F0:0F:00:42:05:12:08:00 SRC=192.168.32.40 DST=192.168.32.45 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8824 DF PROTO=TCP SPT=51312 DPT=80 SEQ=862487553 ACK=0 WINDOW=29200 SYN 
39 7 veth39i0-IN 10/Dec/2018:21:46:21 -0600 policy DROP: IN=fwbr39i0 OUT=fwbr39i0 PHYSIN=fwln39i0 PHYSOUT=veth39i0 MAC=F0:0F:00:69:29:38:F0:0F:00:42:05:12:08:00 SRC=192.168.32.40 DST=192.168.32.40 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=8824 DF PROTO=TCP SPT=51312 DPT=80 SEQ=862487553 ACK=0 WINDOW=29200 SYN

Any help is appreciated!

 

[SillySysAdmin]

For clarification: The machine making the requests is an NGINX server, and the VMs getting hit by the REJECT/DROP policies from other VMs are webservers. I've noticed that the busier web server VMs aren't affected as often as the less busy ones. After a reboot, a REJECT rule will cause a Connection Refused seen by the NGINX server, and the log will show output like the above one, with it hitting three rules from unrelated VMs at the same time. If the server stays busy, the rules won't get tripped.

 

[SillySysAdmin]

https://forum.proxmox.com/threads/r...d-due-to-firewall-from-other-container.37075/

https://bugzilla.proxmox.com/show_bug.cgi?id=1511

It looks like this is directly related to this bug

 

[SillySysAdmin]

# pveversion -v
proxmox-ve: 5.3-1 (running kernel: 4.15.18-9-pve)
pve-manager: 5.3-5 (running version: 5.3-5/97ae681d)
pve-kernel-4.15: 5.2-12
pve-kernel-4.15.18-9-pve: 4.15.18-30
pve-kernel-4.15.17-1-pve: 4.15.17-9
corosync: 2.4.4-pve1
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: 1.2-2
libjs-extjs: 6.0.1-2
libpve-access-control: 5.1-3
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-43
libpve-guest-common-perl: 2.0-18
libpve-http-server-perl: 2.0-11
libpve-storage-perl: 5.0-33
libqb0: 1.0.3-1~bpo9
lvm2: 2.02.168-pve6
lxc-pve: 3.0.2+pve1-5
lxcfs: 3.0.2-2
novnc-pve: 1.0.0-2
proxmox-widget-toolkit: 1.0-22
pve-cluster: 5.0-31
pve-container: 2.0-31
pve-docs: 5.3-1
pve-edk2-firmware: 1.20181023-1
pve-firewall: 3.0-16
pve-firmware: 2.0-6
pve-ha-manager: 2.0-5
pve-i18n: 1.0-9
pve-libspice-server1: 0.14.1-1
pve-qemu-kvm: 2.12.1-1
pve-xtermjs: 1.0-5
qemu-server: 5.0-43
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.12-pve1~bpo1

This does seem to be a pretty severe issue, I am now testing changing my firewall INPUT policies to DROP versus REJECT as suggested in the above bug report to see if this helps with the symptoms.

 

[SillySysAdmin]

Also, any thoughts if an OVS Bridge would work better and also help with this issue versus using just a standard Linux Bridge?

 

[wbumiller]

This is just what happens when using reject rules currently. There is some basic ebtables support, but to fully support such a confguration we also need to add a way to a) use MAC filtering on input and/or b) add a way for users to configure ebtable rules (iow. the ability to add DROP rules for unexpected MAC addresses for the guests, so that they don't reject random packets).