[SOLVED] PVE Firewall not filtering anything

[lucaferr]

Hi! I have a 7 node production cluster with HA and Ceph storage. Every node is running Proxmox 5.4. I recently found out that firewall is not working at all (it was working when we last checked, some months ago): every port of every VM is opened even though per policy should be closed!
It seems like the firewall is disabled, but it is enabled at datacenter level, node level and VM level. Also pve-firewall service seems to be running:

If I run iptables -L I also get a tons of rules...they seem to be the correct rules...
But any VM is accessible on any port, even though the INPUT policy is DROP and no ACCEPT rule is configurated.
Do you have any idea? Please remember that this is a production cluster and I can't afford any downtime...
Thank you!

 

[LnxBil]

You need to enable the firewall on multiple levels:
- on Datacenter level
- on VM-level
- on NIC-VM-level

only if you enable firewall at all three levels, it'll work.

 

[lucaferr]

Yes, I know, and I double checked: it is already enabled at all these levels. Thanks!

 

[spirit]

also check that firewall is enable on vm/ct nics + vm/ct options

and "sysctl -a |grep call":

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

 

[LnxBil]

Yes, I know, and I double checked: it is already enabled at all these levels. Thanks!

and what did you configure to be filtered out? More information please.

 

[lucaferr]

also check that firewall is enable on vm/ct nics + vm/ct options

and "sysctl -a |grep call":

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

I confirm that firewall is enabled on VM's nics (and on their options too). But sysctl returns this instead..what does this mean?
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

 

[lucaferr]

and what did you configure to be filtered out? More information please.

Any rule is completely ignored. For example, a machine with no rules and with a DROP input policy (so every incoming port should be filtered) is instead reachable on any port.

 

[LnxBil]

If I run iptables -L I also get a tons of rules...they seem to be the correct rules...

You have to follow then, e.g. list INPUT, then see where the packages go. PVE created a lot of rules and tables, but you can follow it. Can you post an analysis here?

 

[spirit]

I confirm that firewall is enabled on VM's nics (and on their options too). But sysctl returns this instead..what does this mean?
net.bridge.bridge-nf-call-arptables = 0
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0

Well, filtering occur at bridge level, but use iptables instead ebtables (because we need to filter ip address, ports,..)

This param is some kind of hack in the kernel, to send bridge filtering packets to iptables.

It should be enabled by pve-firewall when it's starting. (you can try to restart pve-firewall). you can also force it in
/etc/sysctl.d/pve.conf.

 

[lucaferr]

Well, filtering occur at bridge level, but use iptables instead ebtables (because we need to filter ip address, ports,..)

This param is some kind of hack in the kernel, to send bridge filtering packets to iptables.

It should be enabled by pve-firewall when it's starting. (you can try to restart pve-firewall). you can also force it in
/etc/sysctl.d/pve.conf.

Thank you so much, we verified and on 6 out of 7 Proxmox nodes the variables net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables were both set to 0. The only node with those variables set to 1 was the only node on which the firewall worked. It was also the node with the lowest uptime (it had an hardware failure, was repaired and rebooted 2 weeks ago). I think that on all the others node, which have an uptime > 200 days, some system updates set those variables to 0.
A pve-firewall restart on all nodes fixed the problem, setting those variables to 1 and finally making the firewall do what it should do.
Thank you so much @spirit !

 

[lucaferr]

Hi all, I re-open this thread after several months because I recently added new nodes to our production cluster (Proxmox VE 5.4-13) and finally figured out what silently disables the firewall (by setting net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables to zero): it's the command pveceph install ! Before the command those variables are 1, after the command they're set to 0. A pve-firewall restart resets them to 1 fixing the problem. It's a very simple fix of a dangerous problem (the firewall seems enabled and working from GUI but in fact it's not!)
I write it here so maybe the Proxmox team can fix the bug (and verify if Proxmox VE 6 is affected too) ;-)

 

[rlljorge]

I have the same issue here, using the version

pve-manager/6.2-11/22fb4983 (running kernel: 5.4.55-1-pve)

But I don't know what command disabled the net.bridge.bridge-nf-call-iptables and net.bridge.bridge-nf-call-ip6tables

This is a CRITICAL SECURITY ISSUE ! Because expose the VMs.

pve-firewall restart is a work around

 

[Kurgan]

Happened to me too. Firewall stopped working for all VMs but was still working for the pve host itself. I don't have ceph. I don't know what made the firewall stop working.

This is A VERY VERY VERY BAD BUG.

I will switch to firewall rules configured inside the VMs.

 

[internbeheer]

Confirmed. Firewall stopped working on a cluster node where the ceph packages were updated but the host wasn't restarted.
No problems on the hosts that were updates and restarted.

This is a CRITICAL SECURITY ISSUE ! Because expose the VMs.

It is.

 

[Kitt]

I am having the same issue. As a temporary workaround, I added an apt rule to reload the pve-firewall any time that packages are updated.

root@proxmox:~# cat /etc/apt/apt.conf.d/99-pve-restart-firewall

# https://forum.proxmox.com/threads/pve-firewall-not-filtering-anything.67084/
# Force restarts the pve-firewall whenever packages are updated.

DPkg::Post-Invoke {"/usr/sbin/pve-firewall restart || true" ; };

I haven't fully tested this since adding it... no guarantees... but if the issue is related to updating Ceph packages then this will reload the firewall afterward.

 

[ph0x]

I have the impression that this is not only related to Ceph packages. My firewall also stops working every few days without any new Ceph packages.

However, it's unacceptable that this does not get any developer attention. Such a bug cannot be accepted in a production grad environment!

 

[Nathan Stratton]

Ran into the same issue, got hacked in the process, having all your firewalls die is NOT a good thing.

 

[t.lamprecht]

I have the impression that this is not only related to Ceph packages. My firewall also stops working every few days without any new Ceph packages.

We have various production and test setups where this is not the case, do you have by any chance any config management (aaltstack, puppet, chef, ...) or other external software running?

 

[ph0x]

I configured ssh and chrony by ansible and sometimes run apt updates through it. But not recently.
Apart from that it's pretty standard install.

 

[Kurgan]

I also have a standard installation, only addition I installed is openvpn that I use for management. Everything else is standard PVE, single host with local LVM storage and no ceph.

When this happens, I see that these values are zero and not 1 as they should:

net.bridge.bridge-nf-call-arptables
net.bridge.bridge-nf-call-ip6tables
net.bridge.bridge-nf-call-iptables

Apart from that, iptables rules are in place. In fact, while firewalling to the vms stops working (because packets from bridges do not traverse iptables anymore) firewalling for the host is still working.