Proxmox VM creation through API | Timeout

[srmvel]

Hi All,

We are trying to create bulk VMs through API. When we choose the VM count as 20 or 40(on API), we are seeing only 14 VMs are getting created, the rest VMs triggers the below error. Please advise on how to solve this?

create full clone of drive ide0 (Template-pool:base-2079-disk-0)
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storageSSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
trying to acquire cfs lock 'storage-SSDPool1' ...
TASK ERROR: clone failed: error during cfs-locked 'storage-SSDPool1' operation: got lock request timeout.

Thank you,
Raj

 

[Stefan_R]

Creating a VM requires a global lock. Are you trying to create all these VMs at once? If so, try waiting for the task to end before creating the next one (or simply add a delay in-between).

 

[srmvel]

Yes right Stefan. We are trying to create 50(example) vms at once. We are trying to achieve it by python script. So I need to set the delay in between on python?
It makes sense?

 

[Stefan_R]

Yes, either that, or (more correctly) query the task status API with the UPID you get back from the 'create' API call (see our API viewer) and wait for status "stopped".

 

[srmvel]

ok thanks, Stefan.I will try that and let you know.

 

[srmvel]

Hi Stefan,

We have set a delay(120 sec) from the python script. Our intention is to create 100 vms using API. When we initiate the script,it will generate the 12 task id's and pass it to Proxmox and then there will a delay of 120 secs before the next schedule(i.e, next 12 VMs creation).

During this process,we still get the below error.
In the first schedule,it created 10 VMs and 2 VMs got failed with the below error.
In the second schedule(after a delay of 120 seconds),it created 2 VMs and 10 VMs got failed with the below error.

trying to acquire cfs lock 'storage-SSDPool1' ...
TASK ERROR: clone failed: error during cfs-locked 'storage-SSDPool1' operation: got lock request timeout.

Please suggest how to solve this issue?

Note:- we are using the test user called " testpmx@pve" to pass the API's.

Thanks,
Raj

 

[Stefan_R]

When we initiate the script,it will generate the 12 task id's and pass it to Proxmox and then there will a delay of 120 secs before the next schedule(i.e, next 12 VMs creation).

No, you can only ever create 1 VM at a time. Create it, then wait for that to finish, then create the second, and so on. This is a limitation of our API, since creating a new VM takes a *global* lock, over the entire cluster.

 

[srmvel]

Thank you Stefan for the update.
Test result 1--- VM creation with 50 GB takes 6 mins to complete through API.
Test result 2 --VM with 100 GB takes 16 mins to complete through API.

As per your response,

If I want to create 20 VMs in a single shot(let's assume 50 GB VM), then I need to set a delay of 6 mins in between(i.e.,one VM creation after other VM).

Roughly, It will take 120 mins to complete all the 20 new VM creations.

Any more advises?

Thank you,
Raj

 

[Stefan_R]

What storage type are you using? 6 minutes per VM seems like a lot...

In general, your idea is correct - however, as stated before, I would rather just ask the API about the task state instead of using a hardcoded timeout.

 

[srmvel]

Hi Stefan, It's a 11 node cluster with ceph storage. We are using crucial ssd disks.We are trying to create all the new VMs from template through API.

It means, the new vm creation should be completed within 2 to 3 mins?

I'm monitoring all these new vm creation from the below commands.

rbd perf image iostat
rbd perf image iotop

What is the accepted write latency, read latency, WR, RD?

Please suggest.

 

[Stefan_R]

Ah, from template, sorry, then it makes sense. I thought you created new empty VMs to then provision them later.

Ceph performance monitoring/tuning is an entirely different topic though, so if you need help with that I'd advise opening a new thread.

 

[srmvel]

ok sure-->I will raise a different thread for Ceph performance monitoring/tuning.

I got confused,If I want to create 20 VMs from template in a single shot(let's assume 50 GB VM), then I need to set a delay of 6 mins in between(i.e.,one VM creation after other VM).

6 mins is ok?

Is this correct approach?

Thanks,
Raj