ebtables-restor consume 100% CPU on proxmox host node

miu

Member
Apr 24, 2021
16
1
8
44
Good day.

I have installed PROXMOX VE on Deb10 host node where has been already running and configured Firewalld & MariaDB & postfix & BIND.

It is new fresh installation of PROXMOX where host is currently w no load, no traffic, yet idling. (host node is OVH KVM 2vCPU, nested virtualization enabled)

Host node net config /etc/network/interfaces:
Code:
auto lo
iface lo inet loopback

# The primary network interface for PROXMOX host node
auto ens3
iface ens3 inet static
        address 152.228.91.75/32
        gateway 152.228.90.1
        pointopoint 152.228.90.1
        dns-nameservers 127.0.0.1 8.8.8.8 8.8.4.4
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up echo 1 > /proc/sys/net/ipv4/conf/ens3/proxy_arp

# for KVM VM-IPs
auto vmbr0
iface vmbr0 inet static
  address 152.228.91.75/32
  bridge_ports none
  bridge_stp off
  bridge_fd 0
#  up ip route for additional-IPs/32 on vmbr0
  up ip route add 178.32.100.221/32 dev vmbr0

Firewalld uses 2 zones:

PUBLIC
for host node (for ens3):
Code:
<?xml version="1.0" encoding="utf-8"?>
<zone>
  <short>Public</short>
  <interface name="ens3"/>
  <service name="smtp"/>
  <service name="smtps"/>
  <service name="pop3"/>
  <service name="pop3s"/>
  <service name="imap"/>
  <service name="imaps"/>
  <port port="587" protocol="tcp"/>
  <port port="53" protocol="tcp"/>
  <port port="2222" protocol="tcp"/>
  <port port="3306" protocol="tcp"/>
  <port port="53" protocol="udp"/>
  <port port="22" protocol="tcp"/>
<port port="8006" protocol="tcp"/>
</zone>

TRUSTED for KVM instance (for vmbr0; allowing all traffic to vmbr0 IPs passthrough host system without filtering by its running firewall):
Code:
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
  <short>Trusted</short>
  <description>All network connections are accepted.</description>
  <interface name="vmbr0"/>
</zone>

ISSUES:
------------
Immediately (or after host node reboot) all works fine:
- Hostnode OS has low CPU load (0% - max 8 or 9%; permanently always under 10%),
- KVM instance (using vmbr0 iface) load is also idling (<4-5% CPU permanently, all time, its LAMP+MTA & all internet connections work fine),
- both no traffic, ALL FINE

BUT: after a few hours (second days morning) i always found:
- Hostnode does have crazy load 100% of CPU: ebtables-restor process consume permanently/whole time all available CPU; traffic is still idling
- (KVM instance remain ok, idling CPU and no traffic too, there is no problem)

BTW: I tried the same (allow all traffic for vbmr0 go directly to VM without be affected by host firewall) also with other (to me know) option with other firewall config as FORWARD rules:
Code:
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -i vmbr0 -j ACCEPT
firewall-cmd --permanent --direct --passthrough ipv4 -I FORWARD -o vmbr0 -j ACCEPT
firewall-cmd --reload; service firewalld restart
But unfortunately, the result after several hours is exactly the same ISSUE as in 1st case with vmbr0 added in TRUSTED zone (host node does have after few hours 100% bussy CPU, consumed by ebtables-restor )

I am already total desparate from it: I killed 2 whole days with this issue (tried all reinstall several times on new, but always exactly the same problem happened and ebtables consume all 100% CPU on idling hostnode & servers. I also cannot find 1 such or similar cases when search on internet or any useful resolutions, suggestions, nothing... Absolutely strange thingh for me and i am not able resolving it and move somewhere.

BTW#2: I know that best is have on host running nothing but Proxmox only, but i need keep current case from severa reasons (have and keep on host node running also firewalld + mentioned software, and cant place it inside nested VM behind proxmox host node)

Thank you very much for all time and help, i would be really very grateful if someone can help me resolve this crazy, strange (for me) issue. All attemps really highly appreciated by me.

regards Milos
 
Last edited:
Additionally - here is actual complete iptables content if this can be useful:

# iptables -S
Code:
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FORWARD_IN_ZONES
-N FORWARD_IN_ZONES_SOURCE
-N FORWARD_OUT_ZONES
-N FORWARD_OUT_ZONES_SOURCE
-N FORWARD_direct
-N FWDI_public
-N FWDI_public_allow
-N FWDI_public_deny
-N FWDI_public_log
-N FWDI_trusted
-N FWDI_trusted_allow
-N FWDI_trusted_deny
-N FWDI_trusted_log
-N FWDO_public
-N FWDO_public_allow
-N FWDO_public_deny
-N FWDO_public_log
-N FWDO_trusted
-N FWDO_trusted_allow
-N FWDO_trusted_deny
-N FWDO_trusted_log
-N INPUT_ZONES
-N INPUT_ZONES_SOURCE
-N INPUT_direct
-N IN_public
-N IN_public_allow
-N IN_public_deny
-N IN_public_log
-N IN_trusted
-N IN_trusted_allow
-N IN_trusted_deny
-N IN_trusted_log
-N OUTPUT_direct
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -j INPUT_direct
-A INPUT -j INPUT_ZONES_SOURCE
-A INPUT -j INPUT_ZONES
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i lo -j ACCEPT
-A FORWARD -j FORWARD_direct
-A FORWARD -j FORWARD_IN_ZONES_SOURCE
-A FORWARD -j FORWARD_IN_ZONES
-A FORWARD -j FORWARD_OUT_ZONES_SOURCE
-A FORWARD -j FORWARD_OUT_ZONES
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -j OUTPUT_direct
-A FORWARD_IN_ZONES -i vmbr0 -j FWDI_trusted
-A FORWARD_IN_ZONES -i ens3 -g FWDI_public
-A FORWARD_IN_ZONES -g FWDI_public
-A FORWARD_OUT_ZONES -o vmbr0 -j FWDO_trusted
-A FORWARD_OUT_ZONES -o ens3 -g FWDO_public
-A FORWARD_OUT_ZONES -g FWDO_public
-A FWDI_public -j FWDI_public_log
-A FWDI_public -j FWDI_public_deny
-A FWDI_public -j FWDI_public_allow
-A FWDI_public -p icmp -j ACCEPT
-A FWDI_trusted -j FWDI_trusted_log
-A FWDI_trusted -j FWDI_trusted_deny
-A FWDI_trusted -j FWDI_trusted_allow
-A FWDI_trusted -j ACCEPT
-A FWDO_public -j FWDO_public_log
-A FWDO_public -j FWDO_public_deny
-A FWDO_public -j FWDO_public_allow
-A FWDO_trusted -j FWDO_trusted_log
-A FWDO_trusted -j FWDO_trusted_deny
-A FWDO_trusted -j FWDO_trusted_allow
-A FWDO_trusted -j ACCEPT
-A INPUT_ZONES -i vmbr0 -j IN_trusted
-A INPUT_ZONES -i ens3 -g IN_public
-A INPUT_ZONES -g IN_public
-A INPUT_direct -p tcp -m multiport --dports 22 -m set --match-set f2b-sshd src -j REJECT --reject-with icmp-port-unreachable
-A IN_public -j IN_public_log
-A IN_public -j IN_public_deny
-A IN_public -j IN_public_allow
-A IN_public -p icmp -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 25 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 465 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 21 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 110 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 995 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 143 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 993 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 587 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 53 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 2222 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p udp -m udp --dport 53 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 3306 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_public_allow -p tcp -m tcp --dport 8006 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
-A IN_trusted -j IN_trusted_log
-A IN_trusted -j IN_trusted_deny
-A IN_trusted -j IN_trusted_allow
-A IN_trusted -j ACCEPT

Thanks for all help!
 
UPDATE:
------------
After another day of searching, reading and investigations probably i found probable reason of problem and also usable solution (not quit clean but yet seems all work well)

CAUSE:
------------
Seems, when firewalld and Proxmox VE meet self on one host/OS, they compete about control of ebtables, have conflict: On begin, ebtables is empty (all connection allowed), but after any time i found ebtable full of nonsensual rules and many times repeating rows, after few hours i found there several thousands of them. Then ibtables-restor try/begin maintain (unsuccessful, around) ebtables and this begin eat permanently crazy portion of CPU resources and cause my issue.

MY PARTIAL SOLUTION (what seems yet that work well):
------------------------------------------------------------------------------------
As i do not need ethernet bridge firewall ever (because each IP is filtered by own OS firewall), i decided try disable ebtables. As i not found option in firewalld DOCS how to do it, i disabled ebtables in kernel (in modprobe.d as blacklisted module). From this time all works well: after more than 24h is load still normal and very low ( CPU load averages example => 0.14 (1 min) 0.21 (5 mins) 0.18 (15 mins) ), when ebtables is off.

My new question is:
-----------------------------
Hopefully disabling of ebtables will not affect any proxmox core functionality and cause any other problems (when i do not need use ebtables firewall ever), or MAY?

If any proxmox staff read this thread and can respond, would be very useful and nice for me when can answer on this question (confirm it), or all opinions, recommendations and suggestions in this direction.

Thank you very much in advance, Regards Milos
 
Hi, you can disable ebtables in gui

->datacenter->firewall->options->ebtables: no
Hello, good note.

I also found later this option in GUI
(but only by chance, because I first look for such an option only under the node because it gave me more logic find it there, not under the data center.)

In case if this disable ebtables in kernel (total) that firewalld cannot find and use ebtables, this can be also possible and very/more simple solution .

Thanks for comment and contribution
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!