pveproxy eats available ram

[Stoiko Ivanov]

Thanks for the update!
the ss output looks like there where tons of pveproxy processes still running - ist this still the case?

also - the anonymization is understandable - but just that I get a clear picture - these were just regular IP-addresses? like e.g. 212.224.123.69

Thanks!

 

[Corwin]

The ss output keeps growing of pveproxy processes every few minutes as long as /etc/default/pveproxy contains at least one IP restriction.
It reverts back to normal behavior when this file is removed, as a workaround, I don't think you need this output isn't it ?
Yes, these are just IPv4 public addresses.

 

[Stoiko Ivanov]

The problem is I cannot reproduce the issue here (I created '/etc/default/pveproxy' like you have it, but with different IP addresses - it works as expected, without any hanging worker processes)...

Could I ask you to provide your (unchanged) '/etc/default/pveproxy' file - if you don't want to share it publicly - you can send it to me via e-mail (s.ivanov _at_ proxmox.com)

I prepared a patch which adds a few debug-statements to the method, which handles the allow_from and deny_form settings - would you be willing to add it in your environment and let pveproxy run with it for a short while?

The file is attached.

* Download it to the server
* Apply it by running:

patch /usr/share/perl5/PVE/APIServer/AnyEvent.pm /path/to/0001-debug-add-debug-log-for-check_host_access.patch.txt
systemctl restart pveproxy

* It will print (quite a bit of) debug information to /var/log/pveproxy/access.log

after you've observed that a few of the hanging worker-processes have accumulated - stop pveproxy, collect the log (and provide it here or via e-mail to me), and reinstall libpve-http-server-perl to get rid of the debug-code:

apt install --reinstall libpve-http-server-perl

Thanks for helping us finding the issue!

 

[Corwin]

just posted...

 

[Stoiko Ivanov]

Thanks! - just took a quick glance - and I think I see a problem in our code - will try to come up with a patch.
It would help to understand how this happens (then I could reproduce it and verify that the patch really solves the issue:
* Do you have some software in place, which connects regularly to 8006 on the box - e.g. for monitoring purposes?
* does anything else open connections and then close them directly?

 

[Corwin]

Ohoh the answer is YES, Monitorix is installed, checking pveproxy process and port on a regular basis, I did not imagine it could harm this way...

 

[Stoiko Ivanov]

I sent a patch for discussion to the list last week (and will try to get it in acceptable shape today):
https://lists.proxmox.com/pipermail/pve-devel/2020-December/046406.html