[SOLVED] PVE Firewall not filtering anything

Jun 21, 2011
64
6
28
Hi! I have a 7 node production cluster with HA and Ceph storage. Every node is running Proxmox 5.4. I recently found out that firewall is not working at all (it was working when we last checked, some months ago): every port of every VM is opened even though per policy should be closed!
It seems like the firewall is disabled, but it is enabled at datacenter level, node level and VM level. Also pve-firewall service seems to be running:
pvefw running.png
If I run iptables -L I also get a tons of rules...they seem to be the correct rules...
But any VM is accessible on any port, even though the INPUT policy is DROP and no ACCEPT rule is configurated.
Do you have any idea? Please remember that this is a production cluster and I can't afford any downtime...
Thank you!
 

LnxBil

Famous Member
Feb 21, 2015
5,482
609
133
Germany
You need to enable the firewall on multiple levels:
- on Datacenter level
- on VM-level
- on NIC-VM-level

only if you enable firewall at all three levels, it'll work.
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
also check that firewall is enable on vm/ct nics + vm/ct options

and "sysctl -a |grep call":

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
 
Jun 21, 2011
64
6
28
also check that firewall is enable on vm/ct nics + vm/ct options

and "sysctl -a |grep call":

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
I confirm that firewall is enabled on VM's nics (and on their options too). But sysctl returns this instead..what does this mean?
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
 

LnxBil

Famous Member
Feb 21, 2015
5,482
609
133
Germany
If I run iptables -L I also get a tons of rules...they seem to be the correct rules...

You have to follow then, e.g. list INPUT, then see where the packages go. PVE created a lot of rules and tables, but you can follow it. Can you post an analysis here?
 

spirit

Famous Member
Apr 2, 2010
4,798
391
103
www.odiso.com
I confirm that firewall is enabled on VM's nics (and on their options too). But sysctl returns this instead..what does this mean?
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

Well, filtering occur at bridge level, but use iptables instead ebtables (because we need to filter ip address, ports,..)

This param is some kind of hack in the kernel, to send bridge filtering packets to iptables.

It should be enabled by pve-firewall when it's starting. (you can try to restart pve-firewall). you can also force it in
/etc/sysctl.d/pve.conf.
 
Jun 21, 2011
64
6
28
Well, filtering occur at bridge level, but use iptables instead ebtables (because we need to filter ip address, ports,..)

This param is some kind of hack in the kernel, to send bridge filtering packets to iptables.

It should be enabled by pve-firewall when it's starting. (you can try to restart pve-firewall). you can also force it in
/etc/sysctl.d/pve.conf.
Thank you so much, we verified and on 6 out of 7 Proxmox nodes the variables net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables were both set to 0. The only node with those variables set to 1 was the only node on which the firewall worked. It was also the node with the lowest uptime (it had an hardware failure, was repaired and rebooted 2 weeks ago). I think that on all the others node, which have an uptime > 200 days, some system updates set those variables to 0.
A pve-firewall restart on all nodes fixed the problem, setting those variables to 1 and finally making the firewall do what it should do.
Thank you so much @spirit !
 
Jun 21, 2011
64
6
28
Hi all, I re-open this thread after several months because I recently added new nodes to our production cluster (Proxmox VE 5.4-13) and finally figured out what silently disables the firewall (by setting net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to zero): it's the command pveceph install ! Before the command those variables are 1, after the command they're set to 0. A pve-firewall restart resets them to 1 fixing the problem. It's a very simple fix of a dangerous problem (the firewall seems enabled and working from GUI but in fact it's not!)
I write it here so maybe the Proxmox team can fix the bug (and verify if Proxmox VE 6 is affected too) ;-)
 

rlljorge

New Member
Jun 2, 2020
9
1
3
40
I have the same issue here, using the version

pve-manager/6.2-11/22fb4983 (running kernel: 5.4.55-1-pve)

But I don't know what command disabled the net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables

This is a CRITICAL SECURITY ISSUE ! Because expose the VMs.

pve-firewall restart is a work around
 
Last edited:
  • Like
Reactions: internbeheer

Kurgan

Member
Apr 27, 2018
28
5
8
51
Happened to me too. Firewall stopped working for all VMs but was still working for the pve host itself. I don't have ceph. I don't know what made the firewall stop working.

This is A VERY VERY VERY BAD BUG.

I will switch to firewall rules configured inside the VMs.
 
  • Like
Reactions: rlljorge
Jul 6, 2020
8
0
1
51
Confirmed. Firewall stopped working on a cluster node where the ceph packages were updated but the host wasn't restarted.
No problems on the hosts that were updates and restarted.

This is a CRITICAL SECURITY ISSUE ! Because expose the VMs.
It is.
 
May 24, 2019
9
1
3
New York, USA
I am having the same issue. As a temporary workaround, I added an apt rule to reload the pve-firewall any time that packages are updated.

Code:
root@proxmox:~# cat /etc/apt/apt.conf.d/99-pve-restart-firewall

# https://forum.proxmox.com/threads/pve-firewall-not-filtering-anything.67084/
# Force restarts the pve-firewall whenever packages are updated.

DPkg::Post-Invoke {"/usr/sbin/pve-firewall restart || true" ; };

I haven't fully tested this since adding it... no guarantees... but if the issue is related to updating Ceph packages then this will reload the firewall afterward.
 
  • Like
Reactions: Kurgan

ph0x

Active Member
Jul 5, 2020
917
144
43
/dev/null
I have the impression that this is not only related to Ceph packages. My firewall also stops working every few days without any new Ceph packages.

However, it's unacceptable that this does not get any developer attention. Such a bug cannot be accepted in a production grad environment!
 

t.lamprecht

Proxmox Staff Member
Staff member
Jul 28, 2015
3,991
941
163
South Tyrol/Italy
shop.maurer-it.com
I have the impression that this is not only related to Ceph packages. My firewall also stops working every few days without any new Ceph packages.
We have various production and test setups where this is not the case, do you have by any chance any config management (aaltstack, puppet, chef, ...) or other external software running?
 

ph0x

Active Member
Jul 5, 2020
917
144
43
/dev/null
I configured ssh and chrony by ansible and sometimes run apt updates through it. But not recently.
Apart from that it's pretty standard install.
 

Kurgan

Member
Apr 27, 2018
28
5
8
51
I also have a standard installation, only addition I installed is openvpn that I use for management. Everything else is standard PVE, single host with local LVM storage and no ceph.

When this happens, I see that these values are zero and not 1 as they should:

net.bridge.bridge-nf-call-arptables
net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-iptables


Apart from that, iptables rules are in place. In fact, while firewalling to the vms stops working (because packets from bridges do not traverse iptables anymore) firewalling for the host is still working.
 
  • Like
Reactions: internbeheer

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!