Slow restore speed on Ceph

[mps209]

Hello Everyone,

Need your expert advise on Ceph with PBS. Please find the below current configuration.

3 Node Ceph Cluster: Xeon Gold 6254 + Ceph Volume (2TB+2TB+4TB - Micron 7450 NVME) each node + 640GB DDR 2999 Mhz + 1GBps NIC for production + 1 GBps NIC for backup and Ceph sync

PBS: 16 Core Xeon E5 + 250 GB DDR4 RAM + 6x4TB MX500 SSD Raid 10 (LVM-Thin) + 1 Gbps Nic.

We also have other Proxmox server running with LVM-Thin Raid 10. While restoring on these LVM-Thin we get around 160 MB/s. While on Ceph cluster we only around 60 MB/s.

Kindly suggest best suitable configuration to get maximum restore speed. During restore and backup I see the NIC usage is just 100 MB/s. Still do I need to upgrade to 10GBps?

Please help.

 

[gurubert]

Writing to Ceph means three network trips instead of just one for LVM.
A 1G network is way too slow for anything related to shared network storage.

 

[mps209]

Writing to Ceph means three network trips instead of just one for LVM.
A 1G network is way too slow for anything related to shared network storage.

I got your point, however we have just 2 VM on each node (this is just a trial) and I dont see ceph nic goes beyond 100MB. So even if there is not much vm and ceph replication load, still 10G is must?

 

[gurubert]

A 1G network interface has a max bandwidth of around 100-120MB/s. This is why even 60MB/s seem to be fast for writing at least for me.

 

[VoIP-Ninja]

Also latency is much lower with 10G than 1G - keep that in mind ( because of 3 writes, commits etc ).