pveproxy eats available ram

Thanks for the update!
the ss output looks like there where tons of pveproxy processes still running - ist this still the case?

also - the anonymization is understandable - but just that I get a clear picture - these were just regular IP-addresses? like e.g. 212.224.123.69

Thanks!
 
The ss output keeps growing of pveproxy processes every few minutes as long as /etc/default/pveproxy contains at least one IP restriction.
It reverts back to normal behavior when this file is removed, as a workaround, I don't think you need this output isn't it ?
Yes, these are just IPv4 public addresses.
 
The problem is I cannot reproduce the issue here (I created '/etc/default/pveproxy' like you have it, but with different IP addresses - it works as expected, without any hanging worker processes)...

Could I ask you to provide your (unchanged) '/etc/default/pveproxy' file - if you don't want to share it publicly - you can send it to me via e-mail (s.ivanov _at_ proxmox.com)

I prepared a patch which adds a few debug-statements to the method, which handles the allow_from and deny_form settings - would you be willing to add it in your environment and let pveproxy run with it for a short while?

The file is attached.

* Download it to the server
* Apply it by running:
Code:
patch /usr/share/perl5/PVE/APIServer/AnyEvent.pm /path/to/0001-debug-add-debug-log-for-check_host_access.patch.txt
systemctl restart pveproxy

* It will print (quite a bit of) debug information to /var/log/pveproxy/access.log

after you've observed that a few of the hanging worker-processes have accumulated - stop pveproxy, collect the log (and provide it here or via e-mail to me), and reinstall libpve-http-server-perl to get rid of the debug-code:
Code:
apt install --reinstall libpve-http-server-perl

Thanks for helping us finding the issue!
 

Attachments

  • 0001-debug-add-debug-log-for-check_host_access.patch.txt
    1.8 KB · Views: 3
Thanks! - just took a quick glance - and I think I see a problem in our code - will try to come up with a patch.
It would help to understand how this happens (then I could reproduce it and verify that the patch really solves the issue:
* Do you have some software in place, which connects regularly to 8006 on the box - e.g. for monitoring purposes?
* does anything else open connections and then close them directly?
 
Ohoh the answer is YES, Monitorix is installed, checking pveproxy process and port on a regular basis, I did not imagine it could harm this way...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!