Memory management: ceph

[lifeboy]

I see that ceph manages memory automatically according to https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#automatic-cache-sizing

Is the following normal for automatic cache sizing then?



Granted, some of the machines have only 8GB of RAM and are used a storage nodes only. I would be happier if the total RAM usage would be below 90%, just to have some headroom...

Anyone who has played with these settings successfully?

 

[spirit]

you need to reduce osd_memory_target.

1GB by osd disk is really the minimum (for hdd it could be ok). But for ssd, you really need something like 3-4GB memory for each osd disk.

 

[lifeboy]

you need to reduce osd_memory_target.

1GB by osd disk is really the minimum (for hdd it could be ok). But for ssd, you really need something like 3-4GB memory for each osd disk.

Thanks!

It seems that luminous doesn't have all the commands to manage this yet. I'm searching the docs now...

I'm systematically upgrading this cluster to the latest version, but I need to understand how to limit the memory usage in the process. Since is just a test and dev cluster, so performance and fast disk speeds are not that important.

 

[KaiS]

Found this on youtube about osd_memory_target:

https://www.youtube.com/watch?v=aaMaMMqOk1o

greetz,
kai

 

[Tmanok]

Please consider that Ceph needs somewhere around:

• 1-2GB/monitor
• 1-2GB/manager
• 3-4GB/OSD

This may also increase per TB by default... You should not be using CEPH with 8GB per machine, this is an enterprise software, I would strongly recommend against <48GB.

Tmanok

 

[lifeboy]

Please consider that Ceph needs somewhere around:

• 1-2GB/monitor
• 1-2GB/manager
• 3-4GB/OSD

This may also increase per TB by default... You should not be using CEPH with 8GB per machine, this is an enterprise software, I would strongly recommend against <48GB.

Tmanok

The beauty of using free/libre software is that one can use it for what one wants. We have a cluster of older repurposed machines that are used as a development and testing environment, be backup to these and it does a great job of it. Surprisingly, the response times are not bad at all.

In the production environment all the nodes have at least 128GB of RAM and newer ones are spec'ed at 512GB.

 

[Tmanok]

The beauty of using free/libre software is that one can use it for what one wants. We have a cluster of older repurposed machines that are used as a development and testing environment, be backup to these and it does a great job of it. Surprisingly, the response times are not bad at all.

In the production environment all the nodes have at least 128GB of RAM and newer ones are spec'ed at 512GB.

Lifeboy, licensing and freedom still clash with reality- those who do not follow requirements will not receive community or professional support.

Anyone can do what they want, but their systems are likely to be unstable, insecure, or poorly performant if they do not meet a product's requirements. Thus they are not supportable- even by the person choosing "freedom" over reality.

Your prod deployment sounds very good btw.


Tmanok