Ceph size looks like shrinking as filling up

G.Geri · Nov 24, 2023

Hello,
Today, we noticed our ceph pool looks like shining as its filling up, is this normal, visual bug, or we need to change something ?

satred with size of 5TB, after puting 1.6TB data on it, looks like its reduced to 3.6TB

Code:

root@pxcl-3:~# ceph status
  cluster:
    id:    
    health: HEALTH_OK

  services:
    mon: 4 daemons, quorum pxcl-1,pxcl-2,pxcl-3,pxcl-4 (age 6d)
    mgr: pxcl-3(active, since 6d), standbys: pxcl-1, pxcl-2, pxcl-4
    osd: 52 osds: 52 up (since 6d), 52 in (since 8d)

  data:
    pools:   2 pools, 129 pgs
    objects: 388.01k objects, 1.5 TiB
    usage:   4.4 TiB used, 16 TiB / 21 TiB avail
    pgs:     129 active+clean

  io:
    client:   8.5 KiB/s rd, 2.1 MiB/s wr, 3 op/s rd, 152 op/s wr

root@pxcl-3:~# ceph osd df tree
ID   CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
 -1         20.66547         -   21 TiB  4.4 TiB  4.4 TiB  223 KiB   85 GiB   16 TiB  21.47  1.00    -          root default
 -3          4.36600         -  4.4 TiB  1.3 TiB  1.3 TiB   66 KiB   22 GiB  3.1 TiB  30.10  1.40    -              host pxcl-1
  0    ssd   0.43660   1.00000  447 GiB  165 GiB  163 GiB    7 KiB  2.2 GiB  282 GiB  36.88  1.72   14      up          osd.0
  1    ssd   0.43660   1.00000  447 GiB  117 GiB  115 GiB    5 KiB  2.5 GiB  330 GiB  26.27  1.22   10      up          osd.1
  2    ssd   0.43660   1.00000  447 GiB  130 GiB  128 GiB    6 KiB  2.5 GiB  317 GiB  29.10  1.35   11      up          osd.2
  3    ssd   0.43660   1.00000  447 GiB   95 GiB   93 GiB    5 KiB  1.9 GiB  352 GiB  21.19  0.99    8      up          osd.3
  4    ssd   0.43660   1.00000  447 GiB  130 GiB  128 GiB    5 KiB  1.7 GiB  317 GiB  29.02  1.35   11      up          osd.4
  5    ssd   0.43660   1.00000  447 GiB  154 GiB  152 GiB   10 KiB  2.4 GiB  293 GiB  34.43  1.60   13      up          osd.5
  6    ssd   0.43660   0.95001  447 GiB  177 GiB  175 GiB    8 KiB  1.9 GiB  270 GiB  39.57  1.84   15      up          osd.6
  7    ssd   0.43660   1.00000  447 GiB  117 GiB  115 GiB   11 KiB  1.9 GiB  330 GiB  26.21  1.22   10      up          osd.7
  8    ssd   0.43660   1.00000  447 GiB  155 GiB  152 GiB    1 KiB  2.4 GiB  292 GiB  34.65  1.61   13      up          osd.8
  9    ssd   0.43660   1.00000  447 GiB  106 GiB  103 GiB    8 KiB  2.5 GiB  341 GiB  23.67  1.10    9      up          osd.9
 -5          4.36600         -  4.4 TiB  1.2 TiB  1.2 TiB   55 KiB   21 GiB  3.1 TiB  28.62  1.33    -              host pxcl-2
 10    ssd   0.43660   1.00000  447 GiB  142 GiB  139 GiB   11 KiB  2.8 GiB  305 GiB  31.68  1.48   12      up          osd.10
 11    ssd   0.43660   1.00000  447 GiB  107 GiB  105 GiB   11 KiB  2.2 GiB  340 GiB  23.86  1.11    9      up          osd.11
 12    ssd   0.43660   1.00000  447 GiB  177 GiB  174 GiB    5 KiB  2.7 GiB  271 GiB  39.48  1.84   15      up          osd.12
 13    ssd   0.43660   1.00000  447 GiB  106 GiB  104 GiB    3 KiB  1.6 GiB  341 GiB  23.68  1.10   10      up          osd.13
 14    ssd   0.43660   1.00000  447 GiB  135 GiB  133 GiB    2 KiB  2.2 GiB  312 GiB  30.20  1.41   11      up          osd.14
 15    ssd   0.43660   1.00000  447 GiB  129 GiB  128 GiB    8 KiB  1.7 GiB  318 GiB  28.93  1.35   11      up          osd.15
 16    ssd   0.43660   1.00000  447 GiB   83 GiB   80 GiB    2 KiB  2.4 GiB  365 GiB  18.48  0.86    7      up          osd.16
 17    ssd   0.43660   0.95001  447 GiB  178 GiB  176 GiB    3 KiB  1.9 GiB  269 GiB  39.80  1.85   15      up          osd.17
 18    ssd   0.43660   1.00000  447 GiB   94 GiB   92 GiB    2 KiB  1.9 GiB  353 GiB  21.08  0.98    8      up          osd.18
 19    ssd   0.43660   1.00000  447 GiB  130 GiB  128 GiB    8 KiB  2.0 GiB  318 GiB  28.99  1.35   11      up          osd.19
 -7          4.36600         -  4.4 TiB  1.3 TiB  1.3 TiB   66 KiB   22 GiB  3.0 TiB  30.57  1.42    -              host pxcl-3
 20    ssd   0.43660   1.00000  447 GiB  164 GiB  162 GiB    6 KiB  2.4 GiB  283 GiB  36.69  1.71   15      up          osd.20
 21    ssd   0.43660   1.00000  447 GiB  164 GiB  161 GiB    5 KiB  2.7 GiB  283 GiB  36.72  1.71   14      up          osd.21
 22    ssd   0.43660   1.00000  447 GiB  141 GiB  139 GiB    7 KiB  2.0 GiB  306 GiB  31.58  1.47   12      up          osd.22
 23    ssd   0.43660   1.00000  447 GiB  142 GiB  139 GiB    8 KiB  2.4 GiB  306 GiB  31.67  1.47   12      up          osd.23
 24    ssd   0.43660   1.00000  447 GiB  176 GiB  173 GiB    3 KiB  2.2 GiB  271 GiB  39.28  1.83   15      up          osd.24
 25    ssd   0.43660   1.00000  447 GiB  141 GiB  140 GiB    7 KiB  1.8 GiB  306 GiB  31.60  1.47   12      up          osd.25
 26    ssd   0.43660   1.00000  447 GiB   96 GiB   94 GiB   11 KiB  1.9 GiB  351 GiB  21.40  1.00    8      up          osd.26
 27    ssd   0.43660   1.00000  447 GiB   95 GiB   92 GiB    8 KiB  2.4 GiB  352 GiB  21.22  0.99    8      up          osd.27
 28    ssd   0.43660   1.00000  447 GiB  142 GiB  139 GiB    9 KiB  2.6 GiB  305 GiB  31.70  1.48   12      up          osd.28
 29    ssd   0.43660   1.00000  447 GiB  107 GiB  105 GiB    2 KiB  1.6 GiB  340 GiB  23.85  1.11    9      up          osd.29
 -9          1.45508         -  1.5 TiB  548 GiB  533 GiB   36 KiB   15 GiB  942 GiB  36.77  1.71    -              host pxcl-4
 30    ssd   0.18188   1.00000  186 GiB   60 GiB   58 GiB    7 KiB  1.8 GiB  126 GiB  32.21  1.50    5      up          osd.30
 31    ssd   0.18188   1.00000  186 GiB   72 GiB   70 GiB    7 KiB  2.4 GiB  114 GiB  38.89  1.81    6      up          osd.31
 32    ssd   0.18188   0.95001  186 GiB   59 GiB   57 GiB    2 KiB  1.9 GiB  127 GiB  31.79  1.48    5      up          osd.32
 33    ssd   0.18188   1.00000  186 GiB   71 GiB   70 GiB    1 KiB  1.6 GiB  115 GiB  38.25  1.78    6      up          osd.33
 34    ssd   0.18188   1.00000  186 GiB   60 GiB   58 GiB    3 KiB  2.1 GiB  126 GiB  32.22  1.50    5      up          osd.34
 35    ssd   0.18188   0.95001  186 GiB  106 GiB  103 GiB    9 KiB  2.5 GiB   81 GiB  56.77  2.64    9      up          osd.35
 36    ssd   0.18188   1.00000  186 GiB   60 GiB   58 GiB    3 KiB  1.5 GiB  127 GiB  32.08  1.49    5      up          osd.36
 37    ssd   0.18188   1.00000  186 GiB   60 GiB   58 GiB    4 KiB  1.8 GiB  127 GiB  31.96  1.49    5      up          osd.37
-11          3.49280         -  3.5 TiB  2.5 GiB  159 MiB      0 B  2.3 GiB  3.5 TiB   0.07  0.00    -              host pxcl-5
 38   ssd2   0.43660   1.00000  447 GiB  318 MiB   20 MiB      0 B  298 MiB  447 GiB   0.07  0.00    0      up          osd.38
 39   ssd2   0.43660   1.00000  447 GiB  310 MiB   20 MiB      0 B  290 MiB  447 GiB   0.07  0.00    0      up          osd.39
 40   ssd2   0.43660   1.00000  447 GiB  322 MiB   20 MiB      0 B  302 MiB  447 GiB   0.07  0.00    0      up          osd.40
 41   ssd2   0.43660   1.00000  447 GiB  310 MiB   20 MiB      0 B  290 MiB  447 GiB   0.07  0.00    0      up          osd.41
 42   ssd2   0.43660   1.00000  447 GiB  322 MiB   20 MiB      0 B  302 MiB  447 GiB   0.07  0.00    0      up          osd.42
 43   ssd2   0.43660   1.00000  447 GiB  322 MiB   20 MiB      0 B  302 MiB  447 GiB   0.07  0.00    0      up          osd.43
 44   ssd2   0.43660   1.00000  447 GiB  314 MiB   20 MiB      0 B  294 MiB  447 GiB   0.07  0.00    0      up          osd.44
 45   ssd2   0.43660   1.00000  447 GiB  314 MiB   20 MiB      0 B  294 MiB  447 GiB   0.07  0.00    0      up          osd.45
-19          2.61960         -  2.6 TiB  1.9 GiB  150 MiB      0 B  1.7 GiB  2.6 TiB   0.07  0.00    -              host pxcl-6
 46   ssd2   0.43660   1.00000  447 GiB  318 MiB   20 MiB      0 B  298 MiB  447 GiB   0.07  0.00    0      up          osd.46
 47   ssd2   0.43660   1.00000  447 GiB  318 MiB   20 MiB      0 B  298 MiB  447 GiB   0.07  0.00    0      up          osd.47
 48   ssd2   0.43660   1.00000  447 GiB  318 MiB   20 MiB      0 B  298 MiB  447 GiB   0.07  0.00    0      up          osd.48
 49   ssd2   0.43660   1.00000  447 GiB  349 MiB   51 MiB      0 B  298 MiB  447 GiB   0.08  0.00    1      up          osd.49
 50   ssd2   0.43660   1.00000  447 GiB  314 MiB   20 MiB      0 B  294 MiB  447 GiB   0.07  0.00    0      up          osd.50
 51   ssd2   0.43660   1.00000  447 GiB  318 MiB   20 MiB      0 B  298 MiB  447 GiB   0.07  0.00    0      up          osd.51
                         TOTAL   21 TiB  4.4 TiB  4.4 TiB  243 KiB   85 GiB   16 TiB  21.47
MIN/MAX VAR: 0.00/2.64  STDDEV: 15.19
root@pxcl-3:~# ceph balancer status
{
    "active": true,
    "last_optimize_duration": "0:00:00.001281",
    "last_optimize_started": "Fri Nov 24 09:50:29 2023",
    "mode": "upmap",
    "no_optimization_needed": true,
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}
root@pxcl-3:~# ceph df detail
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
ssd     15 TiB   10 TiB  4.4 TiB   4.4 TiB      30.43
ssd2   6.1 TiB  6.1 TiB  4.4 GiB   4.4 GiB       0.07
TOTAL   21 TiB   16 TiB  4.4 TiB   4.4 TiB      21.45

--- POOLS ---
POOL    ID  PGS   STORED   (DATA)   (OMAP)  OBJECTS     USED   (DATA)   (OMAP)  %USED  MAX AVAIL  QUOTA OBJECTS  QUOTA BYTES  DIRTY  USED COMPR  UNDER COMPR
.mgr     1    1   31 MiB   31 MiB      0 B        9   93 MiB   93 MiB      0 B      0    2.6 TiB            N/A          N/A    N/A         0 B          0 B
Saturn   2  128  1.4 TiB  1.4 TiB  1.7 KiB  387.77k  4.3 TiB  4.3 TiB  1.7 KiB  43.87    1.9 TiB            N/A          N/A    N/A         0 B          0 B

shanreich · Nov 24, 2023

Your OSDs seem to have a very low number of PGs, which leads to an uneven distribution of data on your OSDs. The maximum capacity of a pool is always based on the fullest OSD in a pool. This is why your pool seems to appear to be shrinking, since the OSDs are filling unevenly.

You can set the target ratio of a pool in the Web UI in order to let the autoscaler increase the number of PGs automatically as needed. You can read more about the autoscaler and how it functions in the Ceph documentation [1].

Please be aware that changing the number of PGs in a pool can lead to high traffic due to rebalancing, which can in turn lead to degraded performance or even interruptions of your service. You can limit the amount of rebalancing traffic, please find a description of this in our wiki [2]. This wiki article is only valid for pools utilizing the mClock scheduler. Please make sure to use the commands from the wiki to check whether you are actually running the mClock scheduler.

[1] https://docs.ceph.com/en/latest/rad...nt-groups/#viewing-pg-scaling-recommendations
[2] https://pve.proxmox.com/wiki/Ceph_mClock_Tuning

shanreich · Nov 24, 2023

Since I only just noticed it: On one node in your ssd pool you have disks of much lower size (186 vs 447 GiB) - so you can store a lot less data there. This also contributes heavily to the problem, since Ceph tries to distribute data evenly between the nodes with the default CRUSH rule. Therefore, it is advisable to have the same amount of space available on each node.

sb-jw · Nov 24, 2023

G.Geri said:
mon: 4 daemons, quorum pxcl-1,pxcl-2,pxcl-3,pxcl-4 (age 6d)
mgr: pxcl-3(active, since 6d), standbys: pxcl-1, pxcl-2, pxcl-4

Please remove the fourth mon and mgr immediately. CEPH needs an odd number for the quorum and with your setup more than 3 mon/mgr is also absolutely oversized.

What is the difference between node 1 - 4 and 5 - 6 that you make different classifications here?

G.Geri · Nov 24, 2023

sb-jw said:
Please remove the fourth mon and mgr immediately. CEPH needs an odd number for the quorum and with your setup more than 3 mon/mgr is also absolutely oversized.

What is the difference between node 1 - 4 and 5 - 6 that you make different classifications here?

1-4 are newer Dell R650
5-6 (and soon 7) is R630

the plan is to use 2 pool on the two server type for the different hardware
5-6(7) currently unused

G.Geri · Nov 24, 2023

shanreich said:
Your OSDs seem to have a very low number of PGs, which leads to an uneven distribution of data on your OSDs. The maximum capacity of a pool is always based on the fullest OSD in a pool. This is why your pool seems to appear to be shrinking, since the OSDs are filling unevenly.

You can set the target ratio of a pool in the Web UI in order to let the autoscaler increase the number of PGs automatically as needed. You can read more about the autoscaler and how it functions in the Ceph documentation [1].

Please be aware that changing the number of PGs in a pool can lead to high traffic due to rebalancing, which can in turn lead to degraded performance or even interruptions of your service. You can limit the amount of rebalancing traffic, please find a description of this in our wiki [2]. This wiki article is only valid for pools utilizing the mClock scheduler. Please make sure to use the commands from the wiki to check whether you are actually running the mClock scheduler.

[1] https://docs.ceph.com/en/latest/rad...nt-groups/#viewing-pg-scaling-recommendations
[2] https://pve.proxmox.com/wiki/Ceph_mClock_Tuning

using Ceph's recommended formula (1266 for 38 osd) i set pg num to 1024

Code:

             (OSDs * 100)
Total PGs =  ------------
              pool size

Re balancing still running but looks like solved the issue.

Thank you.

Search

Search

Ceph size looks like shrinking as filling up

G.Geri

New Member

shanreich

Proxmox Staff Member

shanreich

Proxmox Staff Member

sb-jw

Famous Member

G.Geri

New Member

G.Geri

New Member