[SOLVED] Ceph HDD OSDs start at ~10% usage, 10% lower size

[SteveITS]

I've noticed when creating an OSD on an HDD, it starts out showing about 10% usage, and stays about 10% higher? We are using an SSD for the DB and therefore WAL but it seems like it shouldn't count that...? Also, the size is shown lower in the OSD "details" than the OSD page in PVE.

Screen cap of osd.7 details:

in PVE:


All the SSD only OSDs start at 0%. In this cluster all the SSDs are ~13% used and the few HDDs are all ~23% used.

(and yes we're aware of the bug in 19.2.0, and are recreating them all)

Thanks.

 

[SteveITS]

Example, a newly created OSD (~1 second):


and after about 5 seconds:


This one had a 99 GB DB size on the SSD which (I suspect) is why it's a bit lower than 10%.

 

[SteveITS]

bump

 

[gurubert]

This is normal. Ceph adds the size of the RocksDB device to the total size of the OSD. But since the RocksDB device cannot store any data it is computed as completely used. This is why you see a 10% usage on your fresh OSD.

BTW: 80GB RocksDB device seems a little bit large for a 750GB HDD. And where did you get such small HDDs from?

 

[SteveITS]

Thanks for the reply and info. OK, and it just doesn’t show up on the SSDs because they don’t have a separate DB/WAL. Seems inconsistent but I’ll roll with it. I assumed it was “normal” just didn’t understand why.

These are a few older SAS HDDs we’re carrying forward to Proxmox. They’re 900 GB and I think only one 1.2 TB without looking.

The 10%/80 GB is Proxmox. If you specify the DB drive that’s what it uses by default. I had seen varying recommendations, 4% or 10%, but since it’s generally one small SSD each it doesn’t really matter and I didn’t bother adjusting.

 

[SteveITS]

Just to wrap this up, I had the opportunity to create a new OSD. If I create one on an 900 GB disk without using an SSD I see 838 GB and 0% full:


and if I create one on the same disk using an SSD for WAL it shows as 922 GB and 9.13% full:


Interestingly though the assigned "weight" also changes. I guess that takes into account the 9% full part.