pve-firewall 30% CPU

[Faris Raouf]

Please can someone help me understand what's happening?

On one node in a two-node cluster, I'm seeing pve-firewall show up in "top" using 30% to 40% CPU for a second or so, every 10 seconds or so.

This doesn't happen on the other node.

These are new nodes and barely being used. I have one VE running Asterisk for which I've enabled the Proxmox firewall.

Otherwise, no VEs have a PM firewall enabled.

Cluster-wide, I have the firewall enabled to block access to the nodes themselves (other than to allowed IPs).

There is nothing related that I can see in the syslog, and /var/log/pve-firewall only has short bursts of data from when I enable firewall logging for the VE that has the PM firewall enabled.

This is a powerful, fast system, with almost no load.

restarting the firewall makes no difference (although it does kill my SSH session, for some reason)

What on earth could be causing this?

 

[Faris Raouf]

OK, I think I have a lead on this.

Firstly, it IS happening on both nodes, or at least it is now.

Secondly, in my cluster.fw, I had added a very, very large number of IPs. For example I had created IPSets to block entire countries including China and various others with very large numbers of IP ranges. Simply removing those huge numbers of IPs caused pve-firewalld to use less than 5% CPU rather than 30% to 40%, but still every 10 seconds

Now this, I assume, is to do with some sort of reload or comparison or something to check for changes in the firewall config.
But it still seems like excessive CPU just for that?

I would appreciate other letting me know if they see the same type of "a few %" CPU usage every 10 seconds or so? I now have a very modest number of rules and IPs rather than an excessive list so I would expect to see something similar on other user's nodes.

[EDIT] I'm seeing the same thing on my test cluster. I wouldn't usually use that for comparison because the hardware is old. But I see 15% CPU for a 200k cluster.fw every 10 seconds or so.

 

[spirit]

can you report it to bugzilla.proxmox.com ?

how many ips do you have in the config ?

 

[Faris Raouf]

There are around 27,000 IPs in the file that causes 30 to 40% CPU.

 

[spirit]

I'm not sure, but the code is doing a sort on iplist, maybe it take some time.
(this is needed to have same order than in the real ipset)

maybe could you try to hack the file, remove the sort , relaunch pve-firewall service to compare.
(not in production, as it's possible that ipset will be push again and again, but at least we should see if it's reduce the pve-firewall cpu)



/usr/share/perl5/PVE/Firewall.pm

$ipset_ruleset->{$name} = ["create $name hash:net family $family hashsize $hashsize maxelem $hashsize"];

foreach my $cidr (sort keys %$data) {
my $entry = $data->{$cidr};

my $cmd = "add $name $cidr";
if ($entry->{nomatch}) {
if ($feature_ipset_nomatch) {
push @{$ipset_ruleset->{$name}}, "$cmd nomatch";
} else {
warn "ignore !$cidr - nomatch not supported by kernel\n";
}
} else {
push @{$ipset_ruleset->{$name}}, $cmd;
}
}

 

[hpaluch-pil]

FYI - pve-firewall calls each 10 seconds IP Tables and EB Tables save & restore as can be seen with execsnoop, e.q.

# run on your Proxmox VE Host
apt-get install bpfcc-tools

/usr/sbin/execsnoop-bpfcc -t
# ignore bunch of warnings

Here is output corresponding to pve-firewall every 10 seconds:

TIME(s) PCOMM            PID    PPID   RET ARGS
9.327   iptables-save    3814   1775     0 /sbin/iptables-save
9.330   iptables-restor  3815   1775     0 /sbin/iptables-restore -T filter -n
9.333   ip6tables-save   3816   1775     0 /sbin/ip6tables-save
9.336   ip6tables-resto  3817   1775     0 /sbin/ip6tables-restore -T filter -n
9.339   iptables-save    3818   1775     0 /sbin/iptables-save
9.342   iptables-restor  3819   1775     0 /sbin/iptables-restore -T raw -n
9.345   ip6tables-save   3820   1775     0 /sbin/ip6tables-save
9.348   ip6tables-resto  3821   1775     0 /sbin/ip6tables-restore -T raw -n
9.351   ipset            3822   1775     0 /usr/sbin/ipset save
9.353   ebtables-save    3823   1775     0 /sbin/ebtables-save
9.355   date             3824   3823     0 /bin/date
9.356   sh               3825   3823     0 /bin/sh -c grep -E '^ebtable_' /proc/modules | cut -f1 -d' ' | sed s/ebtable_//
9.356   grep             3826   3825     0 /bin/grep -E ^ebtable_ /proc/modules
9.356   cut              3827   3825     0 /usr/bin/cut -f1 -d
9.356   sed              3828   3825     0 /bin/sed s/ebtable_//
9.357   ebtables         3829   3823     0 /usr/sbin/ebtables -t filter -L
9.360   ebtables-restor  3830   1775     0 /sbin/ebtables-restore

NOTES:
- 1775 is PID of my pve-firewall process
- 3823 is PID of ebtables-save script called from pve-firewall


If you have 27000 IPs in IP Tables or EB Tables it will overload your system every 10 seconds...

My Proxmox VE version is pve-manager/7.1-11/8d529482 (running kernel: 5.13.19-6-pve)

 

[spirit]

Hi,
I have see a bug with ebtables-restore recently, when the list is too big, the ebtables-restore is crashing (seem to be a kernel bug when message is too big).

https://bugzilla.proxmox.com/show_bug.cgi?id=3909

I'll not be fixed by the kernel, the way is to migrate to nftables.
I have begin to work on implementation of nftables but I'll take some time before it's ready to go.

 

[hpaluch-pil]

Yes, but is it really necessary to call such CPU intensive operations ( IP/IP6/EB Tables Save & Restore) every 10 seconds (even on Idle system with no running VM)?

 

[spirit]

Yes, but is it really necessary to call such CPU intensive operations ( IP/IP6/EB Tables Save & Restore) every 10 seconds (even on Idle system with no running VM)?

iptables-save is doing to get current running rules, then the code generate new rules and compare them with current running rules.

(and I think that iptables-save also use a lot of cpu when you have a lot of rules)

restore shouldn't happen if no change exist. (but maybe here, restore is crashing, so it's trying to restore in loop).

They are no "commit" button in proxmox, the daemon always run each 10s.

currently, they are not option to change this 10s value, but
you can hack it, by editing

/usr/share/perl5/PVE/Service/pve_firewall.pm

....
my $updatetime = 10;
...

and restart pve-firewall.



Maybe do a bug report in bugzilla.proxmox.com.
(Maybe it could be possible to cache iptables-save result for example, or optimize some part of the code).