Slow Ceph Speeds

[NightOwl2013]

Hello,

I have really slow Ceph speeds. I don't know what I configured wrong, I could use some help.

I have two nodes in a cluster, using Ceph for storage for VMs. I have both a public and a cluster network. I have configured the ceph config file to see the cluster network, and the OSDs seem to have updated. "attached photos below".

But when I go to use my VMs, storage speed is painfully slow. Copying a 12 GB file on a VM to that VM, I get speeds around 15 MB/s a second.
I know this should be faster, as before adding the second node to the cluster, I had Ceph on just the one, and I was getting around 300 MB/s.
I have tested the cluster network, and it is working as intended, getting 10Gb speeds between the cluster nodes.

Any suggestions?
Thanks!

 

[gurubert]

I wouldn't recommend using Ceph in such a small cluster. Better go with DRBD and LVM on top of that.

You have not described your storage hardware. What kind of NVMe or SSDs do you use?

 

[sb-jw]

What kind of NVMe or SSDs do you use?

And how many of them.

Please also include the settings of the CEPH pool, so is it a replica? If so, is it replica 2/1? Have you adapted the crushmap for this?

 

[NightOwl2013]

I wouldn't recommend using Ceph in such a small cluster. Better go with DRBD and LVM on top of that.

You have not described your storage hardware. What kind of NVMe or SSDs do you use?

Ceph was suggested to me by a college, but I would not be against trying different solutions.

I have added a picture of the OSD configuration to the post if you are interested. Thanks!

 

[sb-jw]

I have added a picture of the OSD configuration to the post if you are interested. Thanks!

I don't see any.

 

[NightOwl2013]

And how many of them.

Please also include the settings of the CEPH pool, so is it a replica? If so, is it replica 2/1? Have you adapted the crushmap for this?

Hello,
I added pictures of the OSD and Pool configurations, along with the crush map in its current state.
Thanks!

 

[NightOwl2013]

I don't see any.

I apologize. It should be there now.

 

[sb-jw]

So, you only have hard drives, which are inherently slow. Then the latency is very high (I currently have no comparison whether it is due to the HDDs themselves or rather in the network) which probably costs the most performance.
For comparison: Enterprise switches and enterprise SSDs have a latency of approximately 1 ms.

Then the ECE02 has 9.86% of data while the ECE01 only has 4.96% of data. Something isn't right and your fill level is higher than it should be. The distribution of PGs is definitely anything but optimal.

You still owe us details about your hardware, you're only sending us pictures here, but no hard facts!

Please post the output of the two commands here in the code tag: ceph osd crush rule dump replicated_rule and ceph -s. And please the VM Config qm config VMID.

 

[NightOwl2013]

So, you only have hard drives, which are inherently slow. Then the latency is very high (I currently have no comparison whether it is due to the HDDs themselves or rather in the network) which probably costs the most performance.
For comparison: Enterprise switches and enterprise SSDs have a latency of approximately 1 ms.

Then the ECE02 has 9.86% of data while the ECE01 only has 4.96% of data. Something isn't right and your fill level is higher than it should be. The distribution of PGs is definitely anything but optimal.

You still owe us details about your hardware, you're only sending us pictures here, but no hard facts!

Please post the output of the two commands here in the code tag: ceph osd crush rule dump replicated_rule and ceph -s. And please the VM Config qm config VMID.

I uploaded the 3 commands to the post. I am converting my servers from Windows server to Proxmox for the first time, so I am a little lacking in experience. Thanks!

 

[gurubert]

Your Ceph cluster will stop working as soon as you lose one of the two nodes.

Never run Ceph with less than three nodes. The MONs have to be an odd number to form a quorum. And min_size=1 will lead to data loss.

HDD only OSDs have a very bad perfomance due to the very random IO pattern of Ceph. Use them at least with the RocksDB on SSD.