[SOLVED] pve-firewall problem

yarii · Apr 29, 2019

I did a reboot yesterday....and then:

Apr 29 18:05:00 v3 systemd[1]: Starting Proxmox VE replication runner...
Apr 29 18:05:01 v3 systemd[1]: Started Proxmox VE replication runner.
Apr 29 18:05:07 v3 pve-firewall[36094]: status update error: unable to apply firewall changes
Apr 29 18:05:16 v3 pve-firewall[36094]: status update error: unable to apply firewall changes
Apr 29 18:05:26 v3 pve-firewall[36094]: status update error: unable to apply firewall changes
Apr 29 18:05:37 v3 pve-firewall[36094]: status update error: unable to apply firewall changes
Apr 29 18:05:46 v3 pve-firewall[36094]: status update error: unable to apply firewall changes
Apr 29 18:05:56 v3 pve-firewall[36094]: status update error: unable to apply firewall changes

Tried:
systemctl restart pve-firewall
but It did not help.

# iptables -L -n | wc -l
872

yarii · Apr 30, 2019

It's somekind of bug.

Does anybody know how to debug it?

Chris · Apr 30, 2019

Hi,
are there any errors listed in the journal? `journalctl -u pve-firewall`
What's the output of `pve-firewall compile`?

yarii · Apr 30, 2019

-- Logs begin at Fri 2019-04-26 17:44:07 CEST, end at Tue 2019-04-30 11:42:08 CEST. --
kwi 26 17:44:21 v3 systemd[1]: Starting Proxmox VE firewall...
kwi 26 17:44:23 v3 pve-firewall[5158]: starting server
kwi 26 17:44:23 v3 systemd[1]: Started Proxmox VE firewall.
kwi 26 17:44:24 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 17:44:33 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 17:44:44 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 17:44:53 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 17:45:03 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 17:45:14 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 17:45:23 v3 pve-firewall[5158]: status update error: unable to apply firewall changes

and this error stays for now in log.

From time to time there is:

kwi 26 23:24:34 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 23:25:28 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 23:25:28 v3 pve-firewall[5158]: firewall update time (44.041 seconds)
kwi 26 23:25:29 v3 pve-firewall[5158]: status update error: unable to apply firewall changes
kwi 26 23:25:38 v3 pve-firewall[5158]: status update error: unable to apply firewall changes

pve-firewall status
Status: enabled/running (pending changes)

pve-firewall compile
there is whole firewall, no errors, all the generated rules are "exists"

only few of the rules are:
exists veth106i0-OUT (CqwbP37DDTKbHiwD6STD141tQno)
-A veth106i0-OUT -p ARP -j veth106i0-OUT-ARP
-A veth106i0-OUT -j ACCEPT
update veth106i0-OUT-ARP (7LohxbDyFGbPns0pWJu/XJkloR0)
-A veth106i0-OUT-ARP -p ARP --arp-ip-src 10.100.0.115/24 -j RETURN
-A veth106i0-OUT-ARP -j DROP

exists veth600i0-OUT (1UxAKreWmw/0v06BeInEzlC5VXo)
-A veth600i0-OUT -s ! 32:39:32:66:39:66 -j DROP
-A veth600i0-OUT -p ARP -j veth600i0-OUT-ARP
-A veth600i0-OUT -j ACCEPT
update veth600i0-OUT-ARP (N/ziFqeaZraGod9eBm/SiVKkkds)
-A veth600i0-OUT-ARP -p ARP --arp-ip-src 46.151.191.151/27 -j RETURN
-A veth600i0-OUT-ARP -j DROP

last 4 lines:
ignore FORWARD (zuQi5YOvmMWiM9zohnQw/qWemOA)
ignore INPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
ignore OUTPUT (2jmj7l5rSw0yVb/vlWAYkK/YBwk)
detected changes

What do pve-firewall do when regenerating rules?

yarii · Apr 30, 2019

If I did:
pve-firewall stop

# iptables -L -n | wc -l
8

and after
pve-firewall start

#iptables -L -n | wc -l
792

and then....

# pve-firewall status
Status: enabled/running (pending changes)

Chris · Apr 30, 2019

yarii said:
What do pve-firewall do when regenerating rules?

It runs /sbin/iptables-restore, /sbin/ipset restore,/sbin/ebtables-restore, ecc. with the new rules found in /var/lib/pve-firewall/*
Afterwards it checks if the rules have been applied correctly, which for some reason seems not to be the case for you. ~~Although `pve-firewall compile` should give you an error in that case .~~ No it does not, only states 'detected changes'. Is there any "unable to update chain ..." in the `pve-firewall compile` output?

yarii · Apr 30, 2019

> Is there any "unable to update chain ..." in the `pve-firewall compile` output?

No.

Is it possible to run pve-firewall in debug/foreground mode?

yarii · Apr 30, 2019

Ok, I've got it:

$ pve-firewall stop; pve-firewall start -debug

ebtables : unable to update chain 'veth104i0-OUT-ARP'
ebtables : unable to update chain 'veth106i0-OUT-ARP'
ebtables : unable to update chain 'veth106i1-OUT-ARP'
ebtables : unable to update chain 'veth167i0-OUT-ARP'
ebtables : unable to update chain 'veth202i3-OUT-ARP'
ebtables : unable to update chain 'veth211i0-OUT-ARP'
ebtables : unable to update chain 'veth306i0-OUT-ARP'
ebtables : unable to update chain 'veth306i1-OUT-ARP'
ebtables : unable to update chain 'veth500i0-OUT-ARP'
ebtables : unable to update chain 'veth555i0-OUT-ARP'
ebtables : unable to update chain 'veth600i0-OUT-ARP'
ebtables : unable to update chain 'veth104i0-OUT-ARP'
ebtables : unable to update chain 'veth106i0-OUT-ARP'
ebtables : unable to update chain 'veth106i1-OUT-ARP'
ebtables : unable to update chain 'veth167i0-OUT-ARP'
ebtables : unable to update chain 'veth202i3-OUT-ARP'
ebtables : unable to update chain 'veth211i0-OUT-ARP'
ebtables : unable to update chain 'veth306i0-OUT-ARP'
ebtables : unable to update chain 'veth306i1-OUT-ARP'
ebtables : unable to update chain 'veth500i0-OUT-ARP'
ebtables : unable to update chain 'veth555i0-OUT-ARP'
ebtables : unable to update chain 'veth600i0-OUT-ARP'
ebtables : unable to update chain 'veth104i0-OUT-ARP'
ebtables : unable to update chain 'veth106i0-OUT-ARP'
ebtables : unable to update chain 'veth106i1-OUT-ARP'
ebtables : unable to update chain 'veth167i0-OUT-ARP'
ebtables : unable to update chain 'veth202i3-OUT-ARP'
ebtables : unable to update chain 'veth211i0-OUT-ARP'
ebtables : unable to update chain 'veth306i0-OUT-ARP'
ebtables : unable to update chain 'veth306i1-OUT-ARP'
ebtables : unable to update chain 'veth500i0-OUT-ARP'
ebtables : unable to update chain 'veth555i0-OUT-ARP'
ebtables : unable to update chain 'veth600i0-OUT-ARP'
ebtables : unable to update chain 'veth104i0-OUT-ARP'
ebtables : unable to update chain 'veth106i0-OUT-ARP'
ebtables : unable to update chain 'veth106i1-OUT-ARP'
ebtables : unable to update chain 'veth167i0-OUT-ARP'
ebtables : unable to update chain 'veth202i3-OUT-ARP'
ebtables : unable to update chain 'veth211i0-OUT-ARP'
ebtables : unable to update chain 'veth306i0-OUT-ARP'
ebtables : unable to update chain 'veth306i1-OUT-ARP'
ebtables : unable to update chain 'veth500i0-OUT-ARP'
ebtables : unable to update chain 'veth555i0-OUT-ARP'
ebtables : unable to update chain 'veth600i0-OUT-ARP'
ebtables : unable to update chain 'veth104i0-OUT-ARP'
ebtables : unable to update chain 'veth106i0-OUT-ARP'
ebtables : unable to update chain 'veth106i1-OUT-ARP'
ebtables : unable to update chain 'veth167i0-OUT-ARP'
ebtables : unable to update chain 'veth202i3-OUT-ARP'
ebtables : unable to update chain 'veth211i0-OUT-ARP'
ebtables : unable to update chain 'veth306i0-OUT-ARP'
ebtables : unable to update chain 'veth306i1-OUT-ARP'
ebtables : unable to update chain 'veth500i0-OUT-ARP'
ebtables : unable to update chain 'veth555i0-OUT-ARP'
ebtables : unable to update chain 'veth600i0-OUT-ARP'

It's this. What is the procedure of update chains by ebtables?

Chris · Apr 30, 2019

Could you try to disable the ebtables filtering in the Datacenter->Firewall, check the output of `ebtables-save` and do the same after re-enabeling it?

Chris · Apr 30, 2019

Also, try to disable and re-enable the MAC filter for one of the CTs in CT->Firewall->Options and check if the corresponding entry in the debug output disappears.

yarii · Apr 30, 2019

After disabling that option:

# ebtables-save
# Generated by ebtables-save v1.0 on wto, 30 kwi 2019, 13:08:48 CEST
*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT

And after enabling the rules are restored

yarii · Apr 30, 2019

The error is generated constantly
Apr 30 13:10:00 v3 systemd[1]: Starting Proxmox VE replication runner...
Apr 30 13:10:01 v3 systemd[1]: Started Proxmox VE replication runner.
Apr 30 13:10:06 v3 pve-firewall[11910]: status update error: unable to apply firewall changes
Apr 30 13:10:16 v3 pve-firewall[11910]: status update error: unable to apply firewall changes
Apr 30 13:10:27 v3 pve-firewall[11910]: status update error: unable to apply firewall changes

yarii · Apr 30, 2019

I need to turn off:
CT--> Firewall --> Options --> IP Filter

And the error disappears. So probably this part of code not suits all cases.

yarii · Apr 30, 2019

The problem in my case is sold.
If You don't want to make a changes in code in this part mark post as [SOLVED].

Chris · Apr 30, 2019

Hmm, trying to reproduce it here. Could you provide the output of /etc/pve/cluster.fw, /etc/pve/firewall/<vmid>.fw for one of the affected CTs, the corresponding ct config `pct config <vmid>` and `pveversion -v`? thx

Chris · Apr 30, 2019

Was able to reproduce, filed a bug report: https://bugzilla.proxmox.com/show_bug.cgi?id=2193

yarii · May 7, 2019

If You can reproduce it thats fine. So I mark this thread as [SOLVED] as it is being procesed by proxmox bugzilla for now.

Search

Search

[SOLVED] pve-firewall problem

yarii

Renowned Member

yarii

Renowned Member

Chris

Proxmox Staff Member

yarii

Renowned Member

yarii

Renowned Member

Chris

Proxmox Staff Member

yarii

Renowned Member

yarii

Renowned Member

Chris

Proxmox Staff Member

Chris

Proxmox Staff Member

yarii

Renowned Member

yarii

Renowned Member

yarii

Renowned Member

yarii

Renowned Member

Chris

Proxmox Staff Member

Chris

Proxmox Staff Member

yarii

Renowned Member

We value your privacy