pvefw-logger spinning 1 CPU after 5.4 update

Discussion in 'Proxmox VE: Networking and Firewall' started by danmac, Apr 18, 2019.

  1. danmac

    danmac New Member

    Joined:
    Dec 6, 2016
    Messages:
    18
    Likes Received:
    0
    So I just caught pvefw-logger using 100% of a single CPU core for the last several hours.

    I had a poke around and it seems to be when either a container or VM is started.

    Restarting the pvefw-logger service results in the errant process having to be killed. This started happening shortly after I installed the last series of updates to that machine, it's running 5.4-3:

    Start-Date: 2019-04-17 00:43:05
    Install: libpve-u2f-server-perl:amd64 (1.0-2, automatic), libu2f-server0:amd64 (1.0.1-3+b1, automatic)
    Upgrade: proxmox-widget-toolkit:amd64 (1.0-24, 1.0-25), libhttp-daemon-perl:amd64 (6.01-1, 6.01-2), libssh2-1:amd64 (1.7.0-1, 1.7.0-1+deb9u1), libpve-access-control:amd64 (5.1-3, 5.1-8), libpve-storage-perl:amd64 (5.0-39, 5.0-41), libwbclient0:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), libsystemd0:amd64 (232-25+deb9u9, 232-25+deb9u11), pve-qemu-kvm:amd64 (2.12.1-2, 2.12.1-3), pve-docs:amd64 (5.3-3, 5.4-2), pve-ha-manager:amd64 (2.0-8, 2.0-9), pve-firewall:amd64 (3.0-18, 3.0-19), udev:amd64 (232-25+deb9u9, 232-25+deb9u11), pve-container:amd64 (2.0-35, 2.0-37), pve-cluster:amd64 (5.0-34, 5.0-36), libudev1:amd64 (232-25+deb9u9, 232-25+deb9u11), samba-libs:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), pve-i18n:amd64 (1.0-9, 1.1-4), pve-xtermjs:amd64 (3.10.1-2, 3.12.0-1), pve-manager:amd64 (5.3-12, 5.4-3), samba-common:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), systemd-sysv:amd64 (232-25+deb9u9, 232-25+deb9u11), libpve-common-perl:amd64 (5.0-48, 5.0-50), libpam-systemd:amd64 (232-25+deb9u9, 232-25+deb9u11), systemd:amd64 (232-25+deb9u9, 232-25+deb9u11), qemu-server:amd64 (5.0-47, 5.0-50), libsmbclient:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), smbclient:amd64 (2:4.5.16+dfsg-1, 2:4.5.16+dfsg-1+deb9u1), libpve-http-server-perl:amd64 (2.0-12, 2.0-13), proxmox-ve:amd64 (5.3-1, 5.4-1)
    End-Date: 2019-04-17 00:44:20


    I haven't prodded it any further at this stage - anyone else seeing this behaviour or any suggestions for how to fix it I'm all ears :)
     
  2. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,247
    Likes Received:
    177
    This is a regression, we're onto it. Basically a bug was there a long time, but wasn't showing as some logging wasn't even enabled correctly, once this was fixed it exposed the old bug as the code path now can be hit.

    It's tracked under: https://bugzilla.proxmox.com/show_bug.cgi?id=2178
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    danmac likes this.
  3. danmac

    danmac New Member

    Joined:
    Dec 6, 2016
    Messages:
    18
    Likes Received:
    0
    Thanks for the info :) I have most VMs / LXCs set to "info" for inbound firewall logging so that seems to match my experience.

    Good luck with the fix, let me know if you need a test subject or anything ;)
     
  4. mira

    mira Proxmox Staff Member
    Staff Member

    Joined:
    Feb 25, 2019
    Messages:
    162
    Likes Received:
    14
    The patch has been applied for now. Should be released with pve-firewall 3.0-20 or higher.
    We always appreciate feedback, so feel free to test it ;)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    danmac likes this.
  5. danmac

    danmac New Member

    Joined:
    Dec 6, 2016
    Messages:
    18
    Likes Received:
    0
    Will do, thanks :) I'm on no-sub but I'll keep an eye on the test repo.
     
  6. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,247
    Likes Received:
    177
    a package with this fix included is available through our pvetest repository as of about an hour ago, see https://pve.proxmox.com/wiki/Package_Repositories for info how to setup, or wehre the URLs are for manual download, if you want to test it earlier.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    danmac likes this.
  7. danmac

    danmac New Member

    Joined:
    Dec 6, 2016
    Messages:
    18
    Likes Received:
    0
    Thanks for the heads up Thomas, can confirm 3.0-20 fixes this issue for me and I have not seen any indications of other issues. Also great work Mira on the patch :)

    For anyone else visiting this thread with the same issue, this is what I did to upgrade my no-subscription box from 3.0-19 from the usual repo to 3.0-20 from the test repo, which resolved the issue for me:

    Downloaded http://download.proxmox.com/debian/...st/binary-amd64/pve-firewall_3.0-20_amd64.deb - basically cherry picked this newer version from the PVE test repo.

    Copied the file to the server using scp but you can use whatever you like, or wget directly from the server itself if you allow it to have internet access.

    Stopped all VMs and LXCs. (just to be on the safe side)

    Run dpkg -i /wherever/you/downloaded/the/file/to/pve-firewall_3.0-20_amd64.deb

    Rebooted the machine. (just to be thorough)

    Thanks again chaps, have a good weekend :)
     
  8. t.lamprecht

    t.lamprecht Proxmox Staff Member
    Staff Member

    Joined:
    Jul 28, 2015
    Messages:
    1,247
    Likes Received:
    177
    Your steps look good in general, just a few notes below

    should not be needed for this.

    using:
    Code:
    apt install ./pve-firewall_3.0-20_amd64.deb
    is a bit more failure proof, as apt checks also if the dependencies are satisfied, etc, dpkg doesn't (it's the lower level tool apt uses), often this then can just be fixed with `apt -f install`, but better to just use apt from the beginning, it's the safer interface for humans :)

    Should not be needed either, at least for this specific update, but it surely didn't hurt also..

    Thanks for testing and providing feedback!
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    danmac likes this.
  9. danmac

    danmac New Member

    Joined:
    Dec 6, 2016
    Messages:
    18
    Likes Received:
    0
    Thanks for taking the time to impart your wisdom - you're right of course I should really have used apt rather than dpkg. Old habits die hard :)

    I did notice during my initial fault-finding that restarting the pve-firewall service was freezing my SSH session (firewall's state table getting cleared maybe) so I was playing it safe with the guests, particularly as some of mine talk amongst themselves. If I remember correctly though, this didn't happen while installing the newer pve-firewall package.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice