inconsistent bridge-nf-call-iptables and resulting issues

cslsa

New Member
Nov 17, 2026
7
0
1
There is one parameter and two opinions from two different proxmox packages on its value:

Code:
# grep -r bridge-nf-call-iptables /usr
/usr/share/perl5/PVE/Firewall.pm:    PVE::ProcFSTools::write_proc_entry("/proc/sys/net/bridge/bridge-nf-call-iptables", "1");
/usr/lib/sysctl.d/10-pve.conf:net.bridge.bridge-nf-call-iptables = 0
# dpkg --search /usr/share/perl5/PVE/Firewall.pm
pve-firewall: /usr/share/perl5/PVE/Firewall.pm
# dpkg --search /usr/lib/sysctl.d/10-pve.conf
pve-cluster: /usr/lib/sysctl.d/10-pve.conf
# md5sum /usr/share/perl5/PVE/Firewall.pm /usr/lib/sysctl.d/10-pve.conf
6e1e8cf7dd107a88fb2446a237651d61  /usr/share/perl5/PVE/Firewall.pm
92a397aceeb6a0a5d3c588949a09521f  /usr/lib/sysctl.d/10-pve.conf
# grep -rE '6e1e8cf7dd107a88fb2446a237651d61|92a397aceeb6a0a5d3c588949a09521f' /var/lib/dpkg/
/var/lib/dpkg/info/pve-cluster.md5sums:92a397aceeb6a0a5d3c588949a09521f  usr/lib/sysctl.d/10-pve.conf
/var/lib/dpkg/info/pve-firewall.md5sums:6e1e8cf7dd107a88fb2446a237651d61  usr/share/perl5/PVE/Firewall.pm

One value is (theoretically) only at boot and the other one from the running firewall (updates it regulary in case someone modified it). However, exceptions apply:

Code:
# grep -r systemd-sysctl /var/lib/dpkg//info/*.postinst
/var/lib/dpkg//info/ceph-osd.postinst:        /lib/systemd/systemd-sysctl /etc/sysctl.d/30-ceph-osd.conf \
/var/lib/dpkg//info/systemd.postinst:    if [ "$(_systemctl show -P LoadState systemd-sysctl.service)" != "masked" ]; then
/var/lib/dpkg//info/systemd.postinst:        _systemctl restart systemd-sysctl.service || true

The systemd posinst hook is actually part of Debian. It is a trigger that is triggered by changes to /usr/lib/sysctl.d. Who triggers it?

Code:
# grep /usr/lib/sysctl.d /var/lib/dpkg/info/*.list | grep pve
/var/lib/dpkg/info/pve-cluster.list:/usr/lib/sysctl.d
/var/lib/dpkg/info/pve-cluster.list:/usr/lib/sysctl.d/10-pve.conf
/var/lib/dpkg/info/pve-container.list:/usr/lib/sysctl.d
/var/lib/dpkg/info/pve-container.list:/usr/lib/sysctl.d/10-pve-ct-inotify-limits.conf
/var/lib/dpkg/info/pve-firewall.list:/usr/lib/sysctl.d
/var/lib/dpkg/info/pve-firewall.list:/usr/lib/sysctl.d/pve-firewall.conf

The effect? The Firewall is off for a moment for bridged interfaces after an proxmox update. Ignoring the other possible implications, this highly impacts the availability of the services in the VM. The default configuration of the firewall drops invalid connections. By not seeing a part of an existing connection or the start of new connections, they are considered invalid as soon as the parameter is back to 1. This breaks connections from the VM to a ceph installation and iSCSI connections from VMs.
 
Last edited:
Some feedback:

We have this issue since years and no one believed that Proxmox is the cause. Everyone just blamed Ceph but no one could find the error there.

I dislike the patch. You could use the override mechanism to shadow /usr/lib/sysctl.d/10-pve.conf and thus avoid reading and applying the wrong value first. Actually, it would be clever to split up that file. The current patch has the potential to only reduce the time window with the wrong value instead of eliminating it.

Some code for manually triggering the issue (if anyone wants to see the effect):

Code:
while true; do echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables; echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables; done
 
I dislike the patch. You could use the override mechanism to shadow /usr/lib/sysctl.d/10-pve.conf and thus avoid reading and applying the wrong value first

systemd-sysctl reads all configuration files first, then resolves the values in order of precedence and only then applies the value - so there is no flip-flopping of the value.
 
If the two nodes ended up on different kernel builds, that alone explains the sysctl difference, worth checking before assuming config drift. bridge-nf-call-iptables also gets flipped by whatever loads first at boot, so a reboot order change can make it inconsistent again. and if any vm has a wireless uplink or a passed-through radio, the bridge hooks behave differently on that path than on plain virtio, so test on the actual interface, not just vmbr0. what does sysctl -a | grep bridge-nf say on each node?
 
@Patrick22 You misunderstand the issue. pve-firewall sets the value to one in a loop repeating all few seconds. As a result, it is consistent after boot - except for Proxmox updates.
 
The proposed patch uses an overlay for sysctl. However, I am not sure whether the write operation into the overlay directory is atomic. Moreover, it calls systemd-sysctl directly instead of using the systemd service unit that would prevent running it from running multiple instances at the same time.
 
Funny enough... I debugged this and this is how I ended up applying the fix with ansible, works everytime now. No more VMs loosing ceph.

YAML:
# pve-cluster's 10-pve.conf turns bridge-nf-call-ip(6)tables off, and pve-firewall turns them
# back on only in its 10 s loop. An upgrade of any package with a file in /usr/lib/sysctl.d
# (pve-cluster, pve-container, pve-firewall, systemd) restarts systemd-sysctl, so the bridged
# traffic skips conntrack for up to 10 s. The busy long-lived TCP flows (the Kubernetes nodes'
# Ceph sessions) then fall outside conntrack's window, PVEFW-FORWARD drops them as INVALID for
# good, and the MDS evicts and blocklists the nodes' CephFS clients after 300 s.
# Sorted after 10-pve.conf, this file wins on every reload.
- name: Bridge firewall kept on across sysctl reloads
  ansible.builtin.copy:
    dest: /etc/sysctl.d/90-pve-firewall-bridge.conf
    mode: "0644"
    content: |
      # Managed by Ansible (ansible/roles/proxmox_host).
      net.bridge.bridge-nf-call-iptables = 1
      net.bridge.bridge-nf-call-ip6tables = 1
  notify: Apply the bridge firewall sysctls

Note: sysctl merges everything first and then only applies the last value based on file ordering name (there's no flapping happening here, 90-pve-firewall-bridge.conf > 10-pve.conf)
 
Last edited:
@romrider While this works for now, it is a manipulation of Proxmox and can break in the future.

The only real fix is Proxmox fixing their system architecture issue - the missing one responsible component for setting this value.