CEPH cluster. Wanted a comment from other PVE ceph users on ram usage per node.

[elimus_]

Currently as all nodes are under load and memory consumption is around 90-95% on each of them.

CEPH cluster details:
* 5 nodes in total, all 5 used for OSD's 3 of them also used as monitors
* All 5 nodes currently have 64G ram
* OSD's 12 disks in total per node - 6x6TB hdd and 6x500G ssd.
* Nodes are running ceph exclusively so no VM's or any other memory consumers.
* PVE v5.3-11

I know that current ceph version in PVE does already support new ram caching for OSD's for up to 4GB per OSD if possible.

This should mean that 48GB is pretty much used up by that OSD caching and 12-14G are left for other processes.

Not sure yet, how ram usage will be affected in this new version when re-balance is started if some OSD or OSD node fails.

PS. Just wanted to know if I have added enough ram and just being paranoid about that 90-95% ram usage per node. Or I should add some more memory?

 

[Daniel S.]

What is your ceph version and are you using bluestore?

Have a look at osd_memory_target which I believe in newest versions it defaults to 4GB.

I do believe that the OSDs should not eat more memory than they are told to, correct me if I'm wrong.

 

[elimus_]

Nodes are still on pve 5.3-11 w/ ceph: 12.2.11-pve1. This version afaik already comes with bluestore ram cache by default

And. Yes, I used the default 4G. For start I didn't see the need to change this.

ceph daemon osd.0 config show | grep memory_target
"osd_memory_target": "4294967296",

Over all I am not worried that It eats most of the ram. I just hope that it wont oom reboot the server cause didn't have enough memory for something that I didn't take in account for planing these nodes as a fairly fresh ceph user.

 

[Alwin]

PS. Just wanted to know if I have added enough ram and just being paranoid about that 90-95% ram usage per node. Or I should add some more memory?

What is the distribution of the memory? As the Linux credo is, free memory is wasted memory.

 

[elimus_]

I think most was eaten up by ceph osd processes. Interestingly I remember seeing them go over that 4294967296 default osd_memory_target, but I will have to check again if it will happen again in current release (pve 5.4-6 | ceph 12.2.12)

I think that I did miscalculate when planing nodes and counted OSD daemons and cache as one and the same and that left me with a
bit to little memory. As these nodes are also without swap. OOM reboot would most likely end bad.

IF I understood ceph docs correctly this time, then I should have that 1G(hdd osd) - 3GB (sdd osd) for daemon + additional space for that bluestore cache 4GB. Before I planned as: ok, 4GB per OSD it is.

Either way 96GB should be ok for these nodes I think.

 

[Alwin]

1G(hdd osd) - 3GB (sdd osd) for daemon

This was replaced by the memory target and counts for the cache, where the OSD tries to stay within. Plus additional memory for each TB of data.
http://docs.ceph.com/docs/luminous/rados/configuration/bluestore-config-ref/#automatic-cache-sizing