Proxmox VM clone delay

[srmvel]

Hi Team,

We see that there is a delay while creating a clone.

It takes more than 8 minutes to take out 100 GB VM cloning.

VM network & Migration network ---> 2 * 10 G NICs.
Ceph Public network and replication network -->2 * 40 G NICs

Where we can see the clone related logs in ceph to start troubleshooting?

I checked here cd /var/log/ceph/ceph .log but nothing is found.

Please advise.

Thank you,
Manivel RR

 

[fiona]

Hi,
does the affected VM also have an EFI disk on Ceph? There was a recent improvement/bugfix, which is available with qemu-server >= 6.3-6.

 

[srmvel]

Hi Fabian,

There is no EFI disk attached to VM.
Any other fix is there?

Thank you,
Raj

 

[fiona]

What kind of disks are you using? Are you cloning from Ceph to Ceph? Does rbd perf image iostat show consistent values during the operation? What about the network traffic? Does enabling krbd (if it's not already) help?

 

[srmvel]

Hi Fabian, Thanks much for your time and your valuable inputs.
We use this 2 TB SSDdisks ct2000mx500ssd1.
We have a CEPH SSD pool and trying to take the clone from the same pool.
We have not enabled krbd under vi /etc/pve/storage.cfg. Is this really needed? Also the client network & replication network is binded with 2 * 40 GB NICS and there should not be any issues.
I see,the wr latency goes between 3 ms to 8ms.



Is this delay triggers because of write latency? What is the accepted write & read latency there?

Also whats the accepted commit latency and apply latency?
ceph osd perf
osd commit_latency(ms) apply_latency(ms)

Any advise are welcome?

Thank you,
Raj

 

[tom]

We use this 2 TB SSDdisks ct2000mx500ssd1

See page number 3 on https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09

(there is a benchmark of a similar crucial SSD)

 

[srmvel]

ok Tom.
It seems,my SSD is not there in PDF.

Is this delay triggers because of write latency? What is the accepted write & read latency there?

Also whats the accepted commit latency and apply latency?
ceph osd perf
osd commit_latency(ms) apply_latency(ms)

Any advise are welcome?

Thank you,
Raj

 

[tom]

It seems,my SSD is not there in PDF.

The PDF shows "Crucial MX100 512GB" - Your Crucial ct2000mx500ssd1 delivers quite similar performance.

 

[srmvel]

ok but the write latency goes above 5 ms while cloning.

 

[tom]

ok but the write latency goes above 5 ms while cloning.

Yes, as you are using a too slow SSDs, not suited for Ceph storage. Read our benchmark paper and follow the recommendations.

https://www.proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09

 

[srmvel]

Ok thanks