pvefw-logger spinning 1 CPU after 5.4 update

[danmac]

So I just caught pvefw-logger using 100% of a single CPU core for the last several hours.

I had a poke around and it seems to be when either a container or VM is started.

Restarting the pvefw-logger service results in the errant process having to be killed. This started happening shortly after I installed the last series of updates to that machine, it's running 5.4-3:

Start-Date: 2019-04-17 00:43:05
Install: libpve-u2f-server-perl:amd64 (1.0-2, automatic), libu2f-server0:amd64 (1.0.1-3+b1, automatic)
Upgrade: proxmox-widget-toolkit:amd64 (1.0-24, 1.0-25), libhttp-daemon-perl:amd64 (6.01-1, 6.01-2), libssh2-1:amd64 (1.7.0-1, 1.7.0-1+deb9u1), libpve-access-control:amd64 (5.1-3, 5.1-8), libpve-storage-perl:amd64 (5.0-39, 5.0-41), libwbclient0:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), libsystemd0:amd64 (232-25+deb9u9, 232-25+deb9u11), pve-qemu-kvm:amd64 (2.12.1-2, 2.12.1-3), pve-docs:amd64 (5.3-3, 5.4-2), pve-ha-manager:amd64 (2.0-8, 2.0-9), pve-firewall:amd64 (3.0-18, 3.0-19), udev:amd64 (232-25+deb9u9, 232-25+deb9u11), pve-container:amd64 (2.0-35, 2.0-37), pve-cluster:amd64 (5.0-34, 5.0-36), libudev1:amd64 (232-25+deb9u9, 232-25+deb9u11), samba-libs:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), pve-i18n:amd64 (1.0-9, 1.1-4), pve-xtermjs:amd64 (3.10.1-2, 3.12.0-1), pve-manager:amd64 (5.3-12, 5.4-3), samba-common:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), systemd-sysv:amd64 (232-25+deb9u9, 232-25+deb9u11), libpve-common-perl:amd64 (5.0-48, 5.0-50), libpam-systemd:amd64 (232-25+deb9u9, 232-25+deb9u11), systemd:amd64 (232-25+deb9u9, 232-25+deb9u11), qemu-server:amd64 (5.0-47, 5.0-50), libsmbclient:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), smbclient:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), libpve-http-server-perl:amd64 (2.0-12, 2.0-13), proxmox-ve:amd64 (5.3-1, 5.4-1)
End-Date: 2019-04-17 00:44:20

I haven't prodded it any further at this stage - anyone else seeing this behaviour or any suggestions for how to fix it I'm all ears

 

[t.lamprecht]

This is a regression, we're onto it. Basically a bug was there a long time, but wasn't showing as some logging wasn't even enabled correctly, once this was fixed it exposed the old bug as the code path now can be hit.

It's tracked under: https://bugzilla.proxmox.com/show_bug.cgi?id=2178

 

[danmac]

Thanks for the info I have most VMs / LXCs set to "info" for inbound firewall logging so that seems to match my experience.

Good luck with the fix, let me know if you need a test subject or anything

 

[mira]

The patch has been applied for now. Should be released with pve-firewall 3.0-20 or higher.
We always appreciate feedback, so feel free to test it

 

[danmac]

Will do, thanks I'm on no-sub but I'll keep an eye on the test repo.

 

[t.lamprecht]

a package with this fix included is available through our pvetest repository as of about an hour ago, see https://pve.proxmox.com/wiki/Package_Repositories for info how to setup, or wehre the URLs are for manual download, if you want to test it earlier.

 

[danmac]

Thanks for the heads up Thomas, can confirm 3.0-20 fixes this issue for me and I have not seen any indications of other issues. Also great work Mira on the patch

For anyone else visiting this thread with the same issue, this is what I did to upgrade my no-subscription box from 3.0-19 from the usual repo to 3.0-20 from the test repo, which resolved the issue for me:

Downloaded http://download.proxmox.com/debian/...st/binary-amd64/pve-firewall_3.0-20_amd64.deb - basically cherry picked this newer version from the PVE test repo.

Copied the file to the server using scp but you can use whatever you like, or wget directly from the server itself if you allow it to have internet access.

Stopped all VMs and LXCs. (just to be on the safe side)

Run dpkg -i /wherever/you/downloaded/the/file/to/pve-firewall_3.0-20_amd64.deb

Rebooted the machine. (just to be thorough)

Thanks again chaps, have a good weekend

 

[t.lamprecht]

Your steps look good in general, just a few notes below

Stopped all VMs and LXCs. (just to be on the safe side)

should not be needed for this.

Run dpkg -i /wherever/you/downloaded/the/file/to/pve-firewall_3.0-20_amd64.deb

using:

apt install ./pve-firewall_3.0-20_amd64.deb

is a bit more failure proof, as apt checks also if the dependencies are satisfied, etc, dpkg doesn't (it's the lower level tool apt uses), often this then can just be fixed with `apt -f install`, but better to just use apt from the beginning, it's the safer interface for humans

Rebooted the machine. (just to be thorough)

Should not be needed either, at least for this specific update, but it surely didn't hurt also..

Thanks for testing and providing feedback!

 

[danmac]

Thanks for taking the time to impart your wisdom - you're right of course I should really have used apt rather than dpkg. Old habits die hard

I did notice during my initial fault-finding that restarting the pve-firewall service was freezing my SSH session (firewall's state table getting cleared maybe) so I was playing it safe with the guests, particularly as some of mine talk amongst themselves. If I remember correctly though, this didn't happen while installing the newer pve-firewall package.