Ceph memory issue

[KeyzerSuze]

Hi


Running a little cluster of beelink mini PC's nice little boxes with nvme drives - they only come with 12G of memory

I have placed 2 4T nvme in there _ there is room for more .

My problem is that OSD memory usage is 24% so that 2 x 24% on a node ... MDS and one other CEPH app take up more memory

I'm running 1 or 2 lXC on each node and 1 VM on the other .. I've hit max memory ....


I want to reconfigure CEPH OSD memory usage down to ~ 500M instead of the 2G+ but I read all of the warnings about it as well.


these are nvme so it should be faster enough ?

any one else pushed down memory usage ?

 

[UdoB]

Running a little cluster of beelink mini PC's nice little boxes with nvme drives -

Yes, I am/was keen to get some of them too.

they only come with 12G of memory

That's really a bummer :-(

I wanted to put two to four OSD in each of it, the actual constrains should allow for four. Now look at https://docs.ceph.com/en/mimic/start/hardware-recommendations/#ram :

• 4 OSD = 4 * 3 to 5 GB = 12 ... 20 GB
• 1 MON = 2 GB
• 1 MGR = 2 GB
• PVE itself = 1 GB

That's 17 ... 25 GB without any LXC or VM (or an MDS)! With only 12 GiB it would swap to death (and slow down to be unusable) - or crash immediately. (Needless to say that to use swap at all would be a cardinal error.)

So... unfortunately with non-upgradable (is it?) RAM this device is just not... useable a good choice for this use case. Now I am seeking the same device with at least 32 GiB Ram...

 

[KeyzerSuze]

Yes, I am/was keen to get some of them too.

That's really a bummer :-(

I wanted to put two to four OSD in each of it, the actual constrains should allow for four. Now look at https://docs.ceph.com/en/mimic/start/hardware-recommendations/#ram :

• 4 OSD = 4 * 3 to 5 GB = 12 ... 20 GB
• 1 MON = 2 GB
• 1 MGR = 2 GB
• PVE itself = 1 GB

That's 17 ... 25 GB without any LXC or VM (or an MDS)! With only 12 GiB it would swap to death (and slow down to be unusable) - or crash immediately. (Needless to say that to use swap at all would be a cardinal error.)

So... unfortunately with non-upgradable (is it?) RAM this device is just not... useable a good choice for this use case. Now I am seeking the same device with at least 32 GiB Ram...

I'm going to move to using ceph from my other cluster - remove the nvme sticks and add them to the other boxes.

Its a shame - posted on their reddit sub... they are coming out with boxs with more memory.

I could also go zfs, but I want the replication syn not async .. so now they become compute nodes