ram usage with bluestore

Stefano Giunchi

Renowned Member
Jan 17, 2016
84
12
73
50
Forlì, Italy
www.soasi.com
this is my test cluster:
node A: n.3 filesystem 1TB OSDs
node B: n.2 filesystem 1TB OSDs, n.1 bluestore 1TB OSDs,
node C: n.6 bluestore 300GB OSDs

I noticed that bluestore OSDs take 3.5GB of RAM each, while the fileystem ones take 0.7 GB each.

Following this thread, I added this to ceph.conf:

Code:
[osd]
bluestore cache size = 1G

Now everything is ok, but is this the right thing to do? Or bluestore OSDs take as much RAM as is available, but releases it if needed?
 
filestore can use the system page cache so the ram use you see is only what the OSD requires to operate.

With bluestore it can’t use the page cache for caching so has its own inbuilt ram cache.

The value you set will be used as an aim to keep the operation ram and the cache ram within target, you will want to set this as close to the default which is 4GB per OSD as you can to allow space for caching.

If 1GB is the best you can do without hitting swap (which will just have a negative impact) then leave it at this.
 
The bluestore_cache_* options are no longer needed. They are replaced
by osd_memory_target, defaulting to 4GB. BlueStore will expand
and contract its cache to attempt to stay within this
limit. Users upgrading should note this is a higher default
than the previous bluestore_cache_size of 1GB, so OSDs using
BlueStore will use more memory by default.
https://ceph.com/releases/v12-2-10-luminous-released/
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!