[SOLVED] mon is low on available space

[kafisatz]

Hi everyone

I am getting a warning:

'HEALTH_WARN mon pve1 is low on available space' although I have about 50% free.
mon.pve1 has 22% avail

Any idea why this happens and what I can do?

Which command would show me the the 22% ?

ceph osd df shows 51% use for all three ssds.....

root@pve1:~# ceph status
  cluster:
    id:     0b05508b-ff81-40c6-a75e-260deb83eabb
    health: HEALTH_WARN
            mon pve1 is low on available space

  services:
    mon: 3 daemons, quorum pve3,pve2,pve1 (age 107m)
    mgr: pve3(active, since 5d), standbys: pve2, pve1
    osd: 3 osds: 3 up (since 5d), 3 in (since 5d)

  data:
    pools:   2 pools, 33 pgs
    objects: 242.92k objects, 947 GiB
    usage:   2.8 TiB used, 2.6 TiB / 5.3 TiB avail
    pgs:     33 active+clean

  io:
    client:   1.1 MiB/s rd, 37 MiB/s wr, 39 op/s rd, 92 op/s wr



root@pve1:~# ceph health
HEALTH_WARN mon pve1 is low on available space


root@pve1:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd    5.3 TiB  2.6 TiB  2.8 TiB   2.8 TiB      51.91
TOTAL  5.3 TiB  2.6 TiB  2.8 TiB   2.8 TiB      51.91

--- POOLS ---
POOL    ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr     5    1  2.7 MiB        2  8.1 MiB      0    785 GiB
cpool2   7   32  943 GiB  242.91k  2.8 TiB  54.56    785 GiB
root@pve1:~#


root@pve1:~# ceph osd df
ID  CLASS  WEIGHT   REWEIGHT  SIZE     RAW USE  DATA     OMAP    META     AVAIL    %USE   VAR   PGS  STATUS
 2    ssd  1.77930   1.00000  1.8 TiB  946 GiB  943 GiB  33 KiB  3.3 GiB  876 GiB  51.92  1.00   33      up
 0    ssd  1.77930   1.00000  1.8 TiB  946 GiB  943 GiB  29 KiB  3.2 GiB  876 GiB  51.91  1.00   33      up
 1    ssd  1.77930   1.00000  1.8 TiB  946 GiB  943 GiB  31 KiB  3.2 GiB  876 GiB  51.91  1.00   33      up
                       TOTAL  5.3 TiB  2.8 TiB  2.8 TiB  94 KiB  9.7 GiB  2.6 TiB  51.91
MIN/MAX VAR: 1.00/1.00  STDDEV: 0.00
root@pve1:~#

 

[VictorSTS]

You should check the free space in the root (/) partition, where the monitor is installed. The variable that sets the warning is mon_data_avail_warn and will pop up that warning if you have less than 30% free space in the disk/partition that hosts /var/lib/ceph/mon. There's another variable mon_data_avail_crit that will show a critical error when 5% or less is available.

 

[kafisatz]

Thanks. You helped me.
I had a temporary file from FIO testing (10GB), thus my root partition had 25GB used from 32GB (indeed only 22% available). I deleted that file now and have 53% free again, which is consistent with what ceph is showing.
I.e. in my case I have a single NVME of 2TB per host. On there proxmox is installed (32GB) and the rest of the drive is used for ceph.