Proxmox API performance issues

[abhishek.singh]

Hi everyone,

I'm having some performance issues with the Proxmox API on my cluster. As the number of users and virtual machines grows, the API becomes slower and slower to respond.

I've already checked that the hardware is top-notch and the servers have enough resources, and I've also checked the Corosync cluster using the pvecm status command. There are no errors or warnings in the output.

I'm not sure what else could be causing the performance issues with the API. Can anyone suggest any other steps I can take to troubleshoot the problem?

I'd really appreciate any suggestions or solutions you might have. Thanks in advance for your help!

 

[bbgeek17]

A performance issue cannot be solved unless it can be defined and quantified.
- What API operations are you referring to specifically?
- Are they generated by a management platform? If they are - what is it?
- What is the number of users and VM when performance is perceived acceptable? How is it measured?
- What is the number of users/VM when the performance is perceived slow? How is it measured?
- Can the situation be easily reproduced? If it can, list the steps.

Have you examined /var/log/pve/pveproxy/access.log - it displays API requests and time it takes to complete each one (last field).

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Matthias.]

That's not quite right.
The last field of /var/log/pveproxy/access.log is the content length. I don't think the duration gets logged anywhere.
Where did you get this information from?

Edit: When I add 1000 VM, the /cluster/resourcesAPI call gets about 0.1s slower. I don't think that's a big problem, so you probably mean something else?

 

[bbgeek17]

That's not quite right.
The last field of /var/log/pveproxy/access.log is the content length. I don't think the duration gets logged anywhere.
Where did you get this information from?

Edit: When I add 1000 VM, the /cluster/resourcesAPI call gets about 0.1s slower. I don't think that's a big problem, so you probably mean something else?

You are correct, my mistake. I confused the format with our log where we do log the time:

2023-01-10T13:13:58.486 DEBUG   HTTP -- 127.0.0.1 - "GET /api/node/node:10000 1.1" 304 - [5.03ms]

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[abhishek.singh]

Hey, thanks for your response,

Edit: When I add 1000 VM, the /cluster/resourcesAPI call gets about 0.1s slower. I don't think that's a big problem, so you probably mean something else?

No that's exactly what I meant, I noticed that as the no of VPS and Users grows the Server API gets slower and slower.
Our custom software design is that a user is assigned to each VM provisioned. That user has VM.PowerMgmt, VM.Console,VM.Audit access roles assigned to it to manage the VPS. Now API calls are made on behalf of the users to manage and monitor their assigned VPS which has now become super slow taking around 40 to 60 seconds to respond.
We experimented with a test cluster where we imported around 500 users from the production and the API became slow to respond. See, I might be wrong here but is there any other way to troubleshoot the same? Please let me know if there are any logs or info required.

 

[Matthias.]

Right, that's slow.
Please answer the questions asked in #2 (https://forum.proxmox.com/threads/proxmox-api-performance-issues.120747/post-524564), especially which API call is slow and if/how to reproduce it.
Additionally, what's the hardware in your PVE?

I tried reproducing, but couldn't find such a major performance performance loss.
My setup was: PVE with 1000 VM, 1000 Users, each user gets VM.PowerMgmt, VM.Console,VM.Audit on one VM.

 

[fabian]

Now API calls are made on behalf of the users to manage and monitor their assigned VPS which has now become super slow taking around 40 to 60 seconds to respond.

the question is really how many and which API calls you make (the pveproxy access log should give you a good overview!). there are some that are really expensive, if you are hammering those the response time will tank..