[SOLVED] PVE Firewall not filtering anything

lucaferr

Renowned Member
Jun 21, 2011
71
8
73
Hi! I have a 7 node production cluster with HA and Ceph storage. Every node is running Proxmox 5.4. I recently found out that firewall is not working at all (it was working when we last checked, some months ago): every port of every VM is opened even though per policy should be closed!
It seems like the firewall is disabled, but it is enabled at datacenter level, node level and VM level. Also pve-firewall service seems to be running:
pvefw running.png
If I run iptables -L I also get a tons of rules...they seem to be the correct rules...
But any VM is accessible on any port, even though the INPUT policy is DROP and no ACCEPT rule is configurated.
Do you have any idea? Please remember that this is a production cluster and I can't afford any downtime...
Thank you!
 
also check that firewall is enable on vm/ct nics + vm/ct options

and "sysctl -a |grep call":

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
I confirm that firewall is enabled on VM's nics (and on their options too). But sysctl returns this instead..what does this mean?
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
 
If I run iptables -L I also get a tons of rules...they seem to be the correct rules...

You have to follow then, e.g. list INPUT, then see where the packages go. PVE created a lot of rules and tables, but you can follow it. Can you post an analysis here?
 
I confirm that firewall is enabled on VM's nics (and on their options too). But sysctl returns this instead..what does this mean?
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

Well, filtering occur at bridge level, but use iptables instead ebtables (because we need to filter ip address, ports,..)

This param is some kind of hack in the kernel, to send bridge filtering packets to iptables.

It should be enabled by pve-firewall when it's starting. (you can try to restart pve-firewall). you can also force it in
/etc/sysctl.d/pve.conf.
 
Well, filtering occur at bridge level, but use iptables instead ebtables (because we need to filter ip address, ports,..)

This param is some kind of hack in the kernel, to send bridge filtering packets to iptables.

It should be enabled by pve-firewall when it's starting. (you can try to restart pve-firewall). you can also force it in
/etc/sysctl.d/pve.conf.
Thank you so much, we verified and on 6 out of 7 Proxmox nodes the variables net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables were both set to 0. The only node with those variables set to 1 was the only node on which the firewall worked. It was also the node with the lowest uptime (it had an hardware failure, was repaired and rebooted 2 weeks ago). I think that on all the others node, which have an uptime > 200 days, some system updates set those variables to 0.
A pve-firewall restart on all nodes fixed the problem, setting those variables to 1 and finally making the firewall do what it should do.
Thank you so much @spirit !
 
Hi all, I re-open this thread after several months because I recently added new nodes to our production cluster (Proxmox VE 5.4-13) and finally figured out what silently disables the firewall (by setting net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to zero): it's the command pveceph install ! Before the command those variables are 1, after the command they're set to 0. A pve-firewall restart resets them to 1 fixing the problem. It's a very simple fix of a dangerous problem (the firewall seems enabled and working from GUI but in fact it's not!)
I write it here so maybe the Proxmox team can fix the bug (and verify if Proxmox VE 6 is affected too) ;-)
 
  • Like
Reactions: pvps1
I have the same issue here, using the version

pve-manager/6.2-11/22fb4983 (running kernel: 5.4.55-1-pve)

But I don't know what command disabled the net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables

This is a CRITICAL SECURITY ISSUE ! Because expose the VMs.

pve-firewall restart is a work around
 
Last edited:
  • Like
Reactions: internbeheer
Happened to me too. Firewall stopped working for all VMs but was still working for the pve host itself. I don't have ceph. I don't know what made the firewall stop working.

This is A VERY VERY VERY BAD BUG.

I will switch to firewall rules configured inside the VMs.
 
  • Like
Reactions: rlljorge
I am having the same issue. As a temporary workaround, I added an apt rule to reload the pve-firewall any time that packages are updated.

Code:
root@proxmox:~# cat /etc/apt/apt.conf.d/99-pve-restart-firewall

# https://forum.proxmox.com/threads/pve-firewall-not-filtering-anything.67084/
# Force restarts the pve-firewall whenever packages are updated.

DPkg::Post-Invoke {"/usr/sbin/pve-firewall restart || true" ; };

I haven't fully tested this since adding it... no guarantees... but if the issue is related to updating Ceph packages then this will reload the firewall afterward.
 
  • Like
Reactions: Kurgan
I have the impression that this is not only related to Ceph packages. My firewall also stops working every few days without any new Ceph packages.

However, it's unacceptable that this does not get any developer attention. Such a bug cannot be accepted in a production grad environment!
 
I have the impression that this is not only related to Ceph packages. My firewall also stops working every few days without any new Ceph packages.
We have various production and test setups where this is not the case, do you have by any chance any config management (aaltstack, puppet, chef, ...) or other external software running?
 
I configured ssh and chrony by ansible and sometimes run apt updates through it. But not recently.
Apart from that it's pretty standard install.
 
I also have a standard installation, only addition I installed is openvpn that I use for management. Everything else is standard PVE, single host with local LVM storage and no ceph.

When this happens, I see that these values are zero and not 1 as they should:

net.bridge.bridge-nf-call-arptables
net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-iptables


Apart from that, iptables rules are in place. In fact, while firewalling to the vms stops working (because packets from bridges do not traverse iptables anymore) firewalling for the host is still working.
 
  • Like
Reactions: internbeheer

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!