Proxmox VM creation through API | Timeout

S

srmvel

Member
Mar 16, 2021
34
0
6
39
Hi All,

We are trying to create bulk VMs through API. When we choose the VM count as 20 or 40(on API), we are seeing only 14 VMs are getting created, the rest VMs triggers the below error. Please advise on how to solve this?

create full clone of drive ide0 (Template-pool:base-2079-disk-0)
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storageSSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
TASK ERROR: clone failed: error during cfs-locked 'storage-SSDPool1' operation: got lock request timeout.

Thank you,
Raj
 
Creating a VM requires a global lock. Are you trying to create all these VMs at once? If so, try waiting for the task to end before creating the next one (or simply add a delay in-between).
 
Yes right Stefan. We are trying to create 50(example) vms at once. We are trying to achieve it by python script. So I need to set the delay in between on python?
It makes sense?
 
Yes, either that, or (more correctly) query the task status API with the UPID you get back from the 'create' API call (see our API viewer) and wait for status "stopped".
 
Hi Stefan,

We have set a delay(120 sec) from the python script. Our intention is to create 100 vms using API. When we initiate the script,it will generate the 12 task id's and pass it to Proxmox and then there will a delay of 120 secs before the next schedule(i.e, next 12 VMs creation).

During this process,we still get the below error.
In the first schedule,it created 10 VMs and 2 VMs got failed with the below error.
In the second schedule(after a delay of 120 seconds),it created 2 VMs and 10 VMs got failed with the below error.

trying to acquire cfs lock 'storage-SSDPool1' ...
TASK ERROR: clone failed: error during cfs-locked 'storage-SSDPool1' operation: got lock request timeout.

Please suggest how to solve this issue?

Note:- we are using the test user called " testpmx@pve" to pass the API's.

Thanks,
Raj
 
Last edited:
srmvel said:
When we initiate the script,it will generate the 12 task id's and pass it to Proxmox and then there will a delay of 120 secs before the next schedule(i.e, next 12 VMs creation).
Click to expand...
No, you can only ever create 1 VM at a time. Create it, then wait for that to finish, then create the second, and so on. This is a limitation of our API, since creating a new VM takes a *global* lock, over the entire cluster.
 
Thank you Stefan for the update.
Test result 1--- VM creation with 50 GB takes 6 mins to complete through API.
Test result 2 --VM with 100 GB takes 16 mins to complete through API.

As per your response,

If I want to create 20 VMs in a single shot(let's assume 50 GB VM), then I need to set a delay of 6 mins in between(i.e.,one VM creation after other VM).

Roughly, It will take 120 mins to complete all the 20 new VM creations.

Any more advises?

Thank you,
Raj
 
What storage type are you using? 6 minutes per VM seems like a lot...

In general, your idea is correct - however, as stated before, I would rather just ask the API about the task state instead of using a hardcoded timeout.
 
Hi Stefan, It's a 11 node cluster with ceph storage. We are using crucial ssd disks.We are trying to create all the new VMs from template through API.

It means, the new vm creation should be completed within 2 to 3 mins?

I'm monitoring all these new vm creation from the below commands.

rbd perf image iostat
rbd perf image iotop

What is the accepted write latency, read latency, WR, RD?

Please suggest.
 
Ah, from template, sorry, then it makes sense. I thought you created new empty VMs to then provision them later.

Ceph performance monitoring/tuning is an entirely different topic though, so if you need help with that I'd advise opening a new thread.
 
ok sure-->I will raise a different thread for Ceph performance monitoring/tuning.

I got confused,If I want to create 20 VMs from template in a single shot(let's assume 50 GB VM), then I need to set a delay of 6 mins in between(i.e.,one VM creation after other VM).

6 mins is ok?

Is this correct approach?

Thanks,
Raj
 
Last edited:
Stefan_R said:
No, you can only ever create 1 VM at a time. Create it, then wait for that to finish, then create the second, and so on. This is a limitation of our API, since creating a new VM takes a *global* lock, over the entire cluster.
Click to expand...
Resurrecting this four year old topic. We have a very similar use case as the OP and I am running into the same issue; before implementing a cluster-level lock in our application for cloning, I wanted to check if this is still the requirement, or if there might be a better way to do this now.

As it is, without adding my own cluster-level lock and polling Proxmox for task completion, linked clones are quick enough such that with the with internal lock retries that Proxmox is doing, 25 or so concurrent requests are succeeding, but when I increase this to 100, I get 4-5 'TASK ERROR: clone failed: cfs-lock 'storage-NFS03' error: got lock request timeout' errors.

I guess I should also ask whether any other VM lifecycle API requests such as delete need the same global application-level lock and task completion check.

Thanks, Stan
 
You must log in or register to reply here.
Top Bottom