[SOLVED] pve-firewall keeps rebuilding due to changing host sort order? [Code changes attached]

[Jhon]

Hi,

On my proxmox server, pve-firewall keeps changing, about every second, from
Status: enabled/running
to
Status: enabled/running (pending changes)
without me making any changes.

Using pve-compile to see what it is doing, the changes I can see are at the bottom in the ebtables part where the clients are sometimes sorted differently and apparently this makes pve-firewall think something has changed and it rebuilds my iptables.

Here is the bottom part of the pve-firewall compile on the first run:

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-OUT (QU31jx1ZjofQ/SzG0V71FIDCtz8)
-A PVEFW-FWBR-OUT -i veth101i0 -j veth101i0-OUT
-A PVEFW-FWBR-OUT -i veth100i0 -j veth100i0-OUT
exists veth100i0-OUT (cQd6Jr2WFd5gbzgLt2EmzR0fvyY)
-A veth100i0-OUT -s ! 26:d4:58:8c:f4:14 -j DROP
-A veth100i0-OUT -j ACCEPT
exists veth101i0-OUT (uDojTOqMjxvVgteCcwfFLBfYXVE)
-A veth101i0-OUT -s ! aa:c0:af:32:6:eb -j DROP
-A veth101i0-OUT -j ACCEPT
delete FORWARD (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)


And here is the second run:

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
update PVEFW-FWBR-OUT (HG0htRKDs6fzs0FF2CPqBxAop/g)
-A PVEFW-FWBR-OUT -i veth100i0 -j veth100i0-OUT
-A PVEFW-FWBR-OUT -i veth101i0 -j veth101i0-OUT
exists veth100i0-OUT (cQd6Jr2WFd5gbzgLt2EmzR0fvyY)
-A veth100i0-OUT -s ! 26:d4:58:8c:f4:14 -j DROP
-A veth100i0-OUT -j ACCEPT
exists veth101i0-OUT (uDojTOqMjxvVgteCcwfFLBfYXVE)
-A veth101i0-OUT -s ! aa:c0:af:32:6:eb -j DROP
-A veth101i0-OUT -j ACCEPT
delete FORWARD (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)

I bolded the part where the sort order of the two clients is changed causing the "difference". This is causing pve-firewall to rewrite my iptables every couple of seconds, which is annoying.

Is there any way to stop this behavior?

I am going to dive in the perl code but it's been a while since I touched perl...

 

[Jhon]

ok, no takers then

I have now found the places in the code where this comes from and have made some changes to the PVE/Firewall.pm file to stop this behavior by making sure that the ebtable rule keys are sorted before the digest is calculated.

This implied creating a duplicate of the iptables_chain_digest sub (I named it ebtables_chain_digest) and adding a "sort" to it. I could not add the sort to the original sub since iprules should not be sorted (there is even a note in the original code). Since the ebtables are currently coming out in a more or less random order, I figure sorting them is not going to make matters any worse.

Then I changed the places where the digests for the ebtables are calculated, calling the new sub.

I also made a small change in the sub that displays the pve-firewall status command output to make sure that the ebtable keys are also sorted there by adding the sort to the key loop..

Here is the diff: (block 3 and 4 are the pve_firewall status command output changes)

# diff ./old/Firewall.pm ./new/Firewall.pm
1789a1790,1800
> #JHON
> sub ebtables_chain_digest {
> my ($rules) = @_;
> my $digest = Digest::SHA->new('sha1');
> foreach my $rule (sort @$rules) { # note: sorted
> $digest->add($rule);
> }
> return $digest->b64digest;
> }
> #JHON
>
1864c1875,1878
< $res->{$chain} = iptables_chain_digest($chains->{$chain});
---
> #JHON
> #$res->{$chain} = iptables_chain_digest($chains->{$chain});
> $res->{$chain} = ebtables_chain_digest($chains->{$chain});
> #JHON
3674c3688
< foreach my $vmid (keys %{$vmdata->{qemu}}) {
---
> foreach my $vmid (sort keys %{$vmdata->{qemu}}) {
3695c3709
< foreach my $vmid (keys %{$vmdata->{lxc}}) {
---
> foreach my $vmid (sort keys %{$vmdata->{lxc}}) {
3859c3873,3876
< my $statushash = get_ruleset_status($ruleset, $active_chains, \&iptables_chain_digest, $verbose);
---
> #JHON
> #my $statushash = get_ruleset_status($ruleset, $active_chains, \&iptables_chain_digest, $verbose);
> my $statushash = get_ruleset_status($ruleset, $active_chains, \&ebtables_chain_digest, $verbose);
> #JHON
4028c4045,4048
< my $ebtables_statushash = get_ruleset_status($ebtables_ruleset, $active_ebtables_chains, \&iptables_chain_digest, 0);
---
> #JHON
> #my $ebtables_statushash = get_ruleset_status($ebtables_ruleset, $active_ebtables_chains, \&iptables_chain_digest, 0);
> my $ebtables_statushash = get_ruleset_status($ebtables_ruleset, $active_ebtables_chains, \&ebtables_chain_digest, 0);
> #JHON


With these changes, the pve-firewall command has been rock-solid for the last hour or so and the resulting ruleset in iptables is identical to one of the two versions it was ping-ponging between earlier.

Hope this helps somebody. o/

 

[Stoiko Ivanov]

Hi,
thanks for the report - it would be great if you open a bugreport in our bugzilla: https://bugzilla.proxmox.com/
Contributions are always more than welcome! - If you'd consider submitting the patch take a look at the developer docs:
https://pve.proxmox.com/wiki/Developer_Documentation
and our contributions page: https://www.proxmox.com/en/proxmox-ve/get-involved

 

[Jhon]

Just a quick note that the code snips above are for the previous version of proxmox. I just upgraded to the most recent version and the problem of course came back.

Attached is a version that works with 5.2.2. (Just search for the JHON label to find the changes.)
Make sure to remove the .txt extension I needed to add to be able to upload it here.

 

[Stoiko Ivanov]

We started working on a fix - hopefully the issue will be resolved in the next release.

 

[Virtualizer]

We have same problem - the firewall hangs - equal with last updates
Linux pm21-host 4.15.18-1-pve #1 SMP PVE 4.15.18-15 (Wed, 04 Jul 2018 15:42:56 +0200) x86_64 GNU/Linux

 

[Stoiko Ivanov]

managed to reproduce the problem and a patch is on the mailing-list - follow the status in our bugzilla:
https://bugzilla.proxmox.com/show_bug.cgi?id=1841

Thanks for the report!