Ceph pool full?

[iwiebe]

I'm stumped, I have 34TB of raw storage left but the Ceph pools are reporting as full. They've hit some sort of arbitrary limit, but I can't see anything in the configuration of the pool that would lead to this state. I do have one OSD of the 21 that hit the full quota, would that cause this?

GLOBAL:
    SIZE     AVAIL      RAW USED     %RAW USED
    152T     35637G         118T         77.23
POOLS:
    NAME             ID     USED       %USED      MAX AVAIL     OBJECTS 
    ceph-vm          8      11926M     100.00             0         3443
    ceph-lxc         9           0          0             0            0
    ceph-mithril     12     40253G     100.00             0     10334649

 

[AlexLup]

Hi,
I usually get this when one osd is down or full. Or the cache is..

check with "ceph osd tree" and maybe a "ceph osd reweight-by-utilization"

 

[iwiebe]

@AlexLup You are a life-saver! "ceph osd reweight-by-utilization" did exactly what I needed.

Follow-up question: Does Ceph consider 85% of raw capacity to be Full? Seems like that's how it's calculating the available space for the pools (factoring in 3/2 replication as well).

 

[Alwin]

152T 35637G 118T 77.23

In any case of a host (or multiple disks) failure, your cluster will be down. You need to keep enough free space available to account for the recovery of your failure domain.

Does Ceph consider 85% of raw capacity to be Full?

It's the 'near full' state and is a high fill level of the disk where it is not guaranteed to recover data on a other disk, when a disks fails.

ceph-vm 8 11926M 100.00 0 3443

A full pool can be reached independently of the global state and needs separate monitoring.