Proxmox networking problems everyday at the same time

[dqq]

Hi,

We use extensively proxmox HTTP API for creating/managing our containers. We do multiple HTTP requests to API 24/7 - approximately 100-200 per minute.
Whole day our service is able to connect to HTTP API and perform needed actions.
However, everyday around 18:30 GTM+1, HTTP API for firewall actions starts responding with 500 HTTP Error code - it does that for same HTTP requests, that it was able to serve couple minutes earlier.

This problem persists from ~60s to ~5 minutes.

Does proxmox perform any timely actions (such as CRON) during this period of time? Is there any reliable way for introspecting proxmox in that matter?

 

[wolfgang]

Hi,

no there is no scheduler related to the pveproxy or pvedaemon.

it does that for same HTTP requests, that it was able to serve couple minutes earlier.

What do this request?

 

[dqq]

@wolfgang thank you for your interest.

Mainly it tries to:
1)update network configuration of CT (add new NIC)
2)Add firewall rules. (Mainly security groups)

What happens:

Everyday script runs 24/7 without issues.
About 18:30 everyday pveproxy starts not responding to requests - we use python library tenacity for retrying requests on exceptions - sometimes after ~15 same retry hits, proxmox accepts request, sometimes it doesn't.

When you look into access.log in proxmox, it says that http status code for those requests was 500, but we have put SSL stripping proxy in the middle, and we can see in proxy traffic that there was no response (ConnectTimeout during waiting for response from proxmox).