ceph osd backfill - some osd's dont appear to be balanced?

[cfgmgr]

Removing the VG just prompted to make the LV go with it (been while since I've had to smoke a VG). In any case I did do a "pvremove /dev/sdg" as there was an LVM signature on the disk. THEN I recreated the osd. New OSD is online and things are starting to move around.

I'm going to pretend its a cake and keep an eye on it since it looks like we're movin data around now:

# ceph osd df tree
ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
-1         36.67854         -   37 TiB   25 TiB   25 TiB  3.2 MiB   68 GiB   11 TiB  69.41  1.00    -          root default
-3         12.22618         -   12 TiB  8.5 TiB  8.4 TiB  1.1 MiB   23 GiB  3.8 TiB  69.25  1.00    -              host hv01
 0    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  118 KiB  3.3 GiB  589 GiB  67.04  0.97   79      up          osd.0
 3    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  110 KiB  3.6 GiB  556 GiB  68.91  0.99   84      up          osd.3
 6    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  103 KiB  3.3 GiB  542 GiB  69.68  1.00   81      up          osd.6
 9    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  115 KiB  3.4 GiB  456 GiB  74.51  1.07   92      up          osd.9
15    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  289 KiB  3.3 GiB  538 GiB  69.93  1.01   87      up          osd.15
16    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  106 KiB  2.8 GiB  562 GiB  68.57  0.99   77      up          osd.16
17    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  243 KiB  3.1 GiB  606 GiB  66.10  0.95   77      up          osd.17
-5         12.22618         -   12 TiB  8.4 TiB  8.3 TiB  1.2 MiB   24 GiB  3.9 TiB  68.35  0.98    -              host hv02
 1    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  110 KiB  3.6 GiB  477 GiB  73.36  1.06   88      up          osd.1
 4    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  106 KiB  3.1 GiB  606 GiB  66.10  0.95   82      up          osd.4
 7    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  272 KiB  3.3 GiB  542 GiB  69.69  1.00   85      up          osd.7
10    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  271 KiB  3.6 GiB  474 GiB  73.48  1.06   85      up          osd.10
14    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  115 KiB  3.5 GiB  559 GiB  68.74  0.99   79      up          osd.14
18    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  242 KiB  3.4 GiB  631 GiB  64.74  0.93   80      up          osd.18
19    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  101 KiB  3.1 GiB  674 GiB  62.33  0.90   78      up          osd.19
-7         12.22618         -   12 TiB  8.6 TiB  8.6 TiB  940 KiB   22 GiB  3.6 TiB  70.64  1.02    -              host hv03
 2    ssd   1.74660   1.00000  1.7 TiB  1.5 TiB  1.5 TiB  303 KiB  3.7 GiB  287 GiB  83.98  1.21   96      up          osd.2
 5    ssd   1.74660   0.85004  1.7 TiB  1.4 TiB  1.4 TiB  130 KiB  3.9 GiB  359 GiB  79.90  1.15   95      up          osd.5
 8    ssd   1.74660   1.00000  1.7 TiB  1.5 TiB  1.5 TiB  123 KiB  3.7 GiB  297 GiB  83.41  1.20   94      up          osd.8
11    ssd   1.74660   1.00000  1.7 TiB  1.4 TiB  1.4 TiB  131 KiB  3.7 GiB  363 GiB  79.72  1.15   92      up          osd.11
12    ssd   1.74660   1.00000  1.7 TiB  1.4 TiB  1.4 TiB  132 KiB  3.1 GiB  394 GiB  77.98  1.12   94      up          osd.12
13    ssd   1.74660   1.00000  1.7 TiB  1.4 TiB  1.4 TiB  121 KiB  3.0 GiB  389 GiB  78.26  1.13   96      up          osd.13
20    ssd   1.74660   1.00000  1.7 TiB  201 GiB  200 GiB      0 B  519 MiB  1.6 TiB  11.23  0.16   10      up          osd.20
                        TOTAL   37 TiB   25 TiB   25 TiB  3.2 MiB   68 GiB   11 TiB  69.41
MIN/MAX VAR: 0.16/1.21  STDDEV: 14.37

# ceph status
  cluster:
    id:     b7565e52-6907-49f9-85b9-526c3ce94676
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum hv01,hv02,hv03 (age 3M)
    mgr: hv01(active, since 3M), standbys: hv02, hv03
    mds: 1/1 daemons up, 2 standby
    osd: 21 osds: 21 up (since 10m), 21 in (since 11m); 90 remapped pgs

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 577 pgs
    objects: 2.21M objects, 8.4 TiB
    usage:   25 TiB used, 11 TiB / 37 TiB avail
    pgs:     317167/6624987 objects misplaced (4.787%)
             487 active+clean
             90  active+remapped+backfilling

  io:
    client:   37 KiB/s rd, 18 MiB/s wr, 2 op/s rd, 299 op/s wr
    recovery: 1.3 GiB/s, 333 objects/s

 

[sb-jw]

What does the balancer say? That doesn't seem to be quite optimal yet. Maybe the ratio is too high?

 

[cfgmgr]

What does the balancer say? That doesn't seem to be quite optimal yet. Maybe the ratio is too high?

# ceph balancer status
{
    "active": true,
    "last_optimize_duration": "0:00:00.003001",
    "last_optimize_started": "Sun Jan 21 14:56:22 2024",
    "mode": "upmap",
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}

Never seen the balancer say much else.

I think it looks quite a bit better than where we started at least:

# ceph osd df tree
ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
-1         36.67854         -   37 TiB   25 TiB   25 TiB  2.3 MiB   70 GiB   12 TiB  68.19  1.00    -          root default
-3         12.22618         -   12 TiB  8.3 TiB  8.3 TiB  785 KiB   23 GiB  3.9 TiB  68.19  1.00    -              host hv01
 0    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  122 KiB  3.3 GiB  598 GiB  66.55  0.98   80      up          osd.0
 3    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  117 KiB  3.4 GiB  571 GiB  68.08  1.00   83      up          osd.3
 6    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  101 KiB  3.5 GiB  554 GiB  69.02  1.01   82      up          osd.6
 9    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  115 KiB  3.6 GiB  490 GiB  72.58  1.06   91      up          osd.9
15    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  106 KiB  3.0 GiB  603 GiB  66.29  0.97   84      up          osd.15
16    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  115 KiB  3.3 GiB  570 GiB  68.11  1.00   78      up          osd.16
17    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  109 KiB  3.2 GiB  595 GiB  66.71  0.98   79      up          osd.17
-5         12.22618         -   12 TiB  8.3 TiB  8.3 TiB  792 KiB   23 GiB  3.9 TiB  68.19  1.00    -              host hv02
 1    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  118 KiB  3.3 GiB  492 GiB  72.49  1.06   87      up          osd.1
 4    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  106 KiB  3.3 GiB  630 GiB  64.77  0.95   81      up          osd.4
 7    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  118 KiB  3.7 GiB  557 GiB  68.85  1.01   84      up          osd.7
10    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.2 TiB  115 KiB  3.3 GiB  506 GiB  71.70  1.05   83      up          osd.10
14    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  111 KiB  2.8 GiB  544 GiB  69.56  1.02   80      up          osd.14
18    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  109 KiB  3.2 GiB  618 GiB  65.44  0.96   81      up          osd.18
19    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  115 KiB  3.3 GiB  635 GiB  64.52  0.95   81      up          osd.19
-7         12.22618         -   12 TiB  8.3 TiB  8.3 TiB  787 KiB   23 GiB  3.9 TiB  68.20  1.00    -              host hv03
 2    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  113 KiB  3.4 GiB  561 GiB  68.63  1.01   81      up          osd.2
 5    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  118 KiB  3.5 GiB  488 GiB  72.74  1.07   88      up          osd.5
 8    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  105 KiB  3.7 GiB  539 GiB  69.87  1.02   84      up          osd.8
11    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  109 KiB  3.3 GiB  616 GiB  65.57  0.96   78      up          osd.11
12    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  113 KiB  3.3 GiB  624 GiB  65.13  0.96   80      up          osd.12
13    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  112 KiB  3.2 GiB  603 GiB  66.28  0.97   82      up          osd.13
20    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  117 KiB  2.9 GiB  551 GiB  69.18  1.01   84      up          osd.20
                        TOTAL   37 TiB   25 TiB   25 TiB  2.3 MiB   70 GiB   12 TiB  68.19
MIN/MAX VAR: 0.95/1.07  STDDEV: 2.57

 

[sb-jw]

I think it looks quite a bit better than where we started at least:

Yes, that's right.

But OSDs 9, 1, 10 and 5 are slightly higher than the others. If I see it correctly, then the difference is < 5% and is therefore below the default ratio of 5% misplaced. If you want to optimize this further, you would have to lower the ratio to 4%.

 

[cfgmgr]

Yes, that's right.

But OSDs 9, 1, 10 and 5 are slightly higher than the others. If I see it correctly, then the difference is < 5% and is therefore below the default ratio of 5% misplaced. If you want to optimize this further, you would have to lower the ratio to 4%.

Would that cause the other OSDs (1, 9, 5, 10) to come down? Not quite sure why those are outliers.

 

[alexskysilk]

Never seen the balancer say much else.

you have three nodes. There isnt really anywhere to balance to. This feature only begins to be useful when you have 5+ nodes.

 

[alexskysilk]

Would that cause the other OSDs (1, 9, 5, 10) to come down? Not quite sure why those are outliers.

osd 20.

 

[cfgmgr]

osd 20.

Got it. Forgot to come back around to this thread but I will consider this a success and look into possibly expanding the cluster (another node, more disk etc).

Appreciate everyones help here!