[SOLVED] pve-firewall keeps rebuilding due to changing host sort order? [Code changes attached]

Jhon

New Member
Jun 20, 2018
3
0
1
54
Hi,

On my proxmox server, pve-firewall keeps changing, about every second, from
Status: enabled/running
to
Status: enabled/running (pending changes)
without me making any changes.

Using pve-compile to see what it is doing, the changes I can see are at the bottom in the ebtables part where the clients are sometimes sorted differently and apparently this makes pve-firewall think something has changed and it rebuilds my iptables.

Here is the bottom part of the pve-firewall compile on the first run:

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
exists PVEFW-FWBR-OUT (QU31jx1ZjofQ/SzG0V71FIDCtz8)
-A PVEFW-FWBR-OUT -i veth101i0 -j veth101i0-OUT
-A PVEFW-FWBR-OUT -i veth100i0 -j veth100i0-OUT
exists veth100i0-OUT (cQd6Jr2WFd5gbzgLt2EmzR0fvyY)
-A veth100i0-OUT -s ! 26:d4:58:8c:f4:14 -j DROP
-A veth100i0-OUT -j ACCEPT
exists veth101i0-OUT (uDojTOqMjxvVgteCcwfFLBfYXVE)
-A veth101i0-OUT -s ! aa:c0:af:32:6:eb -j DROP
-A veth101i0-OUT -j ACCEPT
delete FORWARD (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)


And here is the second run:

ebtables cmdlist:
exists PVEFW-FORWARD (ULtZ6lqjrD/jAKLY+OZo3BbXs9k)
-A PVEFW-FORWARD -p IPv4 -j ACCEPT
-A PVEFW-FORWARD -p IPv6 -j ACCEPT
-A PVEFW-FORWARD -o fwln+ -j PVEFW-FWBR-OUT
update PVEFW-FWBR-OUT (HG0htRKDs6fzs0FF2CPqBxAop/g)
-A PVEFW-FWBR-OUT -i veth100i0 -j veth100i0-OUT
-A PVEFW-FWBR-OUT -i veth101i0 -j veth101i0-OUT
exists veth100i0-OUT (cQd6Jr2WFd5gbzgLt2EmzR0fvyY)
-A veth100i0-OUT -s ! 26:d4:58:8c:f4:14 -j DROP
-A veth100i0-OUT -j ACCEPT
exists veth101i0-OUT (uDojTOqMjxvVgteCcwfFLBfYXVE)
-A veth101i0-OUT -s ! aa:c0:af:32:6:eb -j DROP
-A veth101i0-OUT -j ACCEPT
delete FORWARD (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
delete OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)

I bolded the part where the sort order of the two clients is changed causing the "difference". This is causing pve-firewall to rewrite my iptables every couple of seconds, which is annoying.

Is there any way to stop this behavior?

I am going to dive in the perl code but it's been a while since I touched perl...
 
ok, no takers then :)

I have now found the places in the code where this comes from and have made some changes to the PVE/Firewall.pm file to stop this behavior by making sure that the ebtable rule keys are sorted before the digest is calculated.

This implied creating a duplicate of the iptables_chain_digest sub (I named it ebtables_chain_digest) and adding a "sort" to it. I could not add the sort to the original sub since iprules should not be sorted (there is even a note in the original code). Since the ebtables are currently coming out in a more or less random order, I figure sorting them is not going to make matters any worse. :)

Then I changed the places where the digests for the ebtables are calculated, calling the new sub.

I also made a small change in the sub that displays the pve-firewall status command output to make sure that the ebtable keys are also sorted there by adding the sort to the key loop..

Here is the diff: (block 3 and 4 are the pve_firewall status command output changes)

# diff ./old/Firewall.pm ./new/Firewall.pm
1789a1790,1800
> #JHON
> sub ebtables_chain_digest {
> my ($rules) = @_;
> my $digest = Digest::SHA->new('sha1');
> foreach my $rule (sort @$rules) { # note: sorted
> $digest->add($rule);
> }
> return $digest->b64digest;
> }
> #JHON
>
1864c1875,1878
< $res->{$chain} = iptables_chain_digest($chains->{$chain});
---
> #JHON
> #$res->{$chain} = iptables_chain_digest($chains->{$chain});
> $res->{$chain} = ebtables_chain_digest($chains->{$chain});
> #JHON
3674c3688
< foreach my $vmid (keys %{$vmdata->{qemu}}) {
---
> foreach my $vmid (sort keys %{$vmdata->{qemu}}) {
3695c3709
< foreach my $vmid (keys %{$vmdata->{lxc}}) {
---
> foreach my $vmid (sort keys %{$vmdata->{lxc}}) {
3859c3873,3876
< my $statushash = get_ruleset_status($ruleset, $active_chains, \&iptables_chain_digest, $verbose);
---
> #JHON
> #my $statushash = get_ruleset_status($ruleset, $active_chains, \&iptables_chain_digest, $verbose);
> my $statushash = get_ruleset_status($ruleset, $active_chains, \&ebtables_chain_digest, $verbose);
> #JHON
4028c4045,4048
< my $ebtables_statushash = get_ruleset_status($ebtables_ruleset, $active_ebtables_chains, \&iptables_chain_digest, 0);
---
> #JHON
> #my $ebtables_statushash = get_ruleset_status($ebtables_ruleset, $active_ebtables_chains, \&iptables_chain_digest, 0);
> my $ebtables_statushash = get_ruleset_status($ebtables_ruleset, $active_ebtables_chains, \&ebtables_chain_digest, 0);
> #JHON


With these changes, the pve-firewall command has been rock-solid for the last hour or so and the resulting ruleset in iptables is identical to one of the two versions it was ping-ponging between earlier.

Hope this helps somebody. o/
 
Just a quick note that the code snips above are for the previous version of proxmox. I just upgraded to the most recent version and the problem of course came back.

Attached is a version that works with 5.2.2. (Just search for the JHON label to find the changes.)
Make sure to remove the .txt extension I needed to add to be able to upload it here.
 

Attachments

  • Firewall.pm.txt
    122.2 KB · Views: 2
We started working on a fix - hopefully the issue will be resolved in the next release.
 
We have same problem - the firewall hangs - equal with last updates
Linux pm21-host 4.15.18-1-pve #1 SMP PVE 4.15.18-15 (Wed, 04 Jul 2018 15:42:56 +0200) x86_64 GNU/Linux
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!