[SOLVED] Trying to understand ceph usable space

C

Cha0s

Well-Known Member
Proxmox Subscriber
Feb 9, 2018
33
8
48
Hello,

We run a PVE cluster of 5 nodes with ceph on each node.
Each node has a number of OSDs, each backed by SSDs of various sizes.

A few months ago the OSDs / SSD drives per node were as follows:

Code: 
PVE1
4x  3.49TiB (3.84TB)
5x  1.75TiB (1.92TB)
3x  745GiB  (800GB)

PVE2
4x  3.49TiB (3.84TB)
5x  1.75TiB (1.92TB)
3x  745GiB  (800GB)

PVE3
4x  3.49TiB (3.84TB)
4x  1.75TiB (1.92TB)
6x  745GiB  (800GB)

PVE4
2x  3.49TiB (3.84TB)
11x 1.75TiB (1.92TB)

PVE5
2x  3.49TiB (3.84TB)
11x 1.75TiB (1.92TB)

The cluster has only one ceph pool configured with the default 3 replicas and 512PGs.
With the above disk configuration the total usable disk space as calculated by ceph was 23.8TiB at the time.

Since the pool's utilization was getting close to 70+% we decided to add more disks.

On all five nodes we added an extra 6.99TiB SSD per node which amounted to 34.95TiB extra raw space and a theoretical 11.65TiB of extra usable space.

So afterwards the disk configuration became as follows:

Code: 
PVE1
4x  3.49TiB (3.84TB)
5x  1.75TiB (1.92TB)
3x  745GiB  (800GB)
1x  6.99TiB (7.68TB)

PVE2
4x  3.49TiB (3.84TB)
5x  1.75TiB (1.92TB)
3x  745GiB  (800GB)
1x  6.99TiB (7.68TB)

PVE3
4x  3.49TiB (3.84TB)
4x  1.75TiB (1.92TB)
6x  745GiB  (800GB)
1x  6.99TiB (7.68TB)

PVE4
2x  3.49TiB (3.84TB)
11x 1.75TiB (1.92TB)
1x  6.99TiB (7.68TB)

PVE5
2x  3.49TiB (3.84TB)
11x 1.75TiB (1.92TB)
1x  6.99TiB (7.68TB)

But after ceph did its rebalancing the total usable space increased only by 4.3TiB, ie 28.1TiB total usable space.
ceph-usable-space.png

Since we are using various sizes of disks, I suspect that the usable space might not be as simple as "raw disks / 3 replicas = usable space", but after installing the 5 new disks of same size I hoped that we it would have given us an increase of ~11TiB.
Obviously there's something I am missing in the way the usable space is calculated.

Could someone more experienced with ceph, shed some light as to why we only have 28.1TiB usable space over 162.73TiB of raw space, which divided by 3 it should theoretically give us ~54.24TiB of usable space?
I get that there are some losses due to overheads, metadata, etc, but we get almost half the usable space!


Would re-arranging the existing SSDs per node yield higher usable space?
If so, could someone provide some insight as to how we should go about it?


Let me know if you need me to provide you with any command results from ceph or PVE, if it will help you understand the situation better.

Thanks!
 
Last edited:
Can you post the output of ceph osd df tree?

If the data is balanced very unevenly, it is possible that a few OSDs are considerably more full than others. That also plays a role in the estimate of usable space.

And while we are at it, for completeness’ sake, please also the output of pveceph pool ls --noborder. Either run it in a wide terminal window or redirect the output into a file to catch it all.
 
Code: 
root@pve01 ~ # ceph osd df tree
ID   CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA      OMAP     META     AVAIL     %USE   VAR   PGS  STATUS  TYPE NAME     
 -1         162.40396         -  131 TiB   53 TiB    53 TiB  1.0 GiB  149 GiB    77 TiB  40.71  1.00    -          root default 
 -3          31.86797         -   32 TiB   13 TiB    13 TiB  291 MiB   40 GiB    19 TiB  41.14  1.01    -              host pve01
  0    ssd    3.49199   1.00000  3.5 TiB  1.9 TiB   1.9 TiB  162 MiB  4.7 GiB   1.6 TiB  54.30  1.33   56      up          osd.0
  1    ssd    3.49199   1.00000  3.5 TiB  1.4 TiB   1.3 TiB   11 MiB  3.5 GiB   2.1 TiB  38.68  0.95   39      up          osd.1
  2    ssd    3.49309   1.00000  3.5 TiB  1.6 TiB   1.6 TiB   16 MiB  4.7 GiB   1.9 TiB  44.69  1.10   45      up          osd.2
  3    ssd    3.49309   1.00000  3.5 TiB  1.6 TiB   1.5 TiB   14 MiB  4.9 GiB   1.9 TiB  44.44  1.09   45      up          osd.3
 12    ssd    1.74599   1.00000  1.7 TiB  854 GiB   852 GiB  6.5 MiB  2.4 GiB   934 GiB  47.79  1.17   24      up          osd.12
 13    ssd    1.74599   1.00000  1.7 TiB  531 GiB   528 GiB   13 MiB  2.2 GiB   1.2 TiB  29.67  0.73   15      up          osd.13
 14    ssd    1.74599   1.00000  1.7 TiB  711 GiB   708 GiB  9.3 MiB  2.3 GiB   1.1 TiB  39.74  0.98   20      up          osd.14
 16    ssd    1.74599   1.00000  1.7 TiB  853 GiB   851 GiB   10 MiB  2.5 GiB   935 GiB  47.73  1.17   24      up          osd.16
 26    ssd    0.72699   1.00000  745 GiB  213 GiB   211 GiB  8.2 MiB  1.4 GiB   532 GiB  28.53  0.70    6      up          osd.26
 27    ssd    0.72699   1.00000  745 GiB  283 GiB   282 GiB  5.7 MiB  1.5 GiB   462 GiB  38.04  0.93    8      up          osd.27
 28    ssd    0.72699   1.00000  745 GiB  283 GiB   281 GiB  4.8 MiB  1.6 GiB   462 GiB  37.94  0.93    8      up          osd.28
 38    ssd    1.74660   1.00000  1.7 TiB  636 GiB   635 GiB  9.7 MiB  1.7 GiB   1.1 TiB  35.58  0.87   18      up          osd.38
 64    ssd    6.98630   1.00000  7.0 TiB  2.5 TiB   2.5 TiB   21 MiB  6.6 GiB   4.5 TiB  35.63  0.88   72      up          osd.64
 -5          31.86577         -      0 B      0 B       0 B      0 B      0 B       0 B      0     0    -              host pve02
  4    ssd    3.49199         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.4
  5    ssd    3.49199         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.5
  6    ssd    3.49199         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.6
  7    ssd    3.49199         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.7
 17    ssd    1.74599         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.17
 18    ssd    1.74660         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.18
 19    ssd    1.74599         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.19
 20    ssd    1.74599         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.20
 21    ssd    1.74599         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.21
 29    ssd    0.72699         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.29
 30    ssd    0.72699         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.30
 31    ssd    0.72699         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.31
 65    ssd    6.98630         0      0 B      0 B       0 B      0 B      0 B       0 B      0     0    0    down          osd.65
 -7          32.30014         -   32 TiB   14 TiB    14 TiB  272 MiB   40 GiB    18 TiB  43.10  1.06    -              host pve03
  8    ssd    3.49199   1.00000  3.5 TiB  1.4 TiB   1.4 TiB  7.8 MiB  3.7 GiB   2.1 TiB  40.84  1.00   41      up          osd.8
  9    ssd    3.49199   1.00000  3.5 TiB  1.5 TiB   1.5 TiB  9.2 MiB  4.1 GiB   2.0 TiB  43.63  1.07   44      up          osd.9
 10    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB   1.2 TiB  7.6 MiB  3.9 GiB   2.2 TiB  35.63  0.88   36      up          osd.10
 11    ssd    3.49199   1.00000  3.5 TiB  1.1 TiB   1.1 TiB  6.8 MiB  3.2 GiB   2.4 TiB  30.72  0.75   31      up          osd.11
 22    ssd    1.74599   1.00000  1.7 TiB  849 GiB   846 GiB  6.6 MiB  2.6 GiB   939 GiB  47.48  1.17   24      up          osd.22
 23    ssd    1.74599   1.00000  1.7 TiB  604 GiB   602 GiB   12 MiB  2.2 GiB   1.2 TiB  33.77  0.83   17      up          osd.23
 24    ssd    1.74599   1.00000  1.7 TiB  1.0 TiB  1022 GiB  8.5 MiB  3.0 GiB   763 GiB  57.30  1.41   29      up          osd.24
 25    ssd    1.74599   1.00000  1.7 TiB  496 GiB   494 GiB  6.8 MiB  1.9 GiB   1.3 TiB  27.75  0.68   14      up          osd.25
 32    ssd    0.72699   1.00000  745 GiB  357 GiB   355 GiB  7.0 MiB  1.4 GiB   388 GiB  47.85  1.18   10      up          osd.32
 33    ssd    0.72699   1.00000  745 GiB  534 GiB   533 GiB  4.0 MiB  1.4 GiB   211 GiB  71.67  1.76   15      up          osd.33
 34    ssd    0.72699   1.00000  745 GiB  389 GiB   388 GiB  4.0 MiB  1.1 GiB   356 GiB  52.22  1.28   11      up          osd.34
 35    ssd    0.72699   1.00000  745 GiB  568 GiB   567 GiB  6.3 MiB  1.4 GiB   177 GiB  76.27  1.87   16      up          osd.35
 36    ssd    0.72699   1.00000  745 GiB  422 GiB   421 GiB  153 MiB  1.2 GiB   323 GiB  56.67  1.39   13      up          osd.36
 37    ssd    0.72699   1.00000  745 GiB  390 GiB   389 GiB  5.4 MiB  1.1 GiB   355 GiB  52.40  1.29   11      up          osd.37
 66    ssd    6.98630   1.00000  7.0 TiB  3.2 TiB   3.1 TiB   27 MiB  7.5 GiB   3.8 TiB  45.15  1.11   91      up          osd.66
 -9          33.18504         -   33 TiB   13 TiB    13 TiB  140 MiB   34 GiB    20 TiB  39.69  0.97    -              host pve04
 15    ssd    3.49309   1.00000  3.5 TiB  1.5 TiB   1.5 TiB  9.6 MiB  3.6 GiB   2.0 TiB  42.49  1.04   43      up          osd.15
 39    ssd    3.49309   1.00000  3.5 TiB  1.5 TiB   1.5 TiB   17 MiB  3.4 GiB   2.0 TiB  43.45  1.07   44      up          osd.39
 40    ssd    1.74660   1.00000  1.7 TiB  739 GiB   737 GiB  6.3 MiB  2.1 GiB   1.0 TiB  41.34  1.02   21      up          osd.40
 41    ssd    1.74660   1.00000  1.7 TiB  674 GiB   673 GiB  8.4 MiB  1.8 GiB   1.1 TiB  37.71  0.93   19      up          osd.41
 42    ssd    1.74660   1.00000  1.7 TiB  746 GiB   744 GiB   11 MiB  2.0 GiB   1.0 TiB  41.73  1.03   21      up          osd.42
 43    ssd    1.74660   1.00000  1.7 TiB  710 GiB   708 GiB   11 MiB  1.9 GiB   1.1 TiB  39.71  0.98   20      up          osd.43
 44    ssd    1.74660   1.00000  1.7 TiB  884 GiB   882 GiB  6.1 MiB  2.1 GiB   905 GiB  49.41  1.21   25      up          osd.44
 45    ssd    1.74660   1.00000  1.7 TiB  674 GiB   672 GiB  7.7 MiB  2.0 GiB   1.1 TiB  37.71  0.93   19      up          osd.45
 46    ssd    1.74660   1.00000  1.7 TiB  496 GiB   495 GiB  8.7 MiB  1.5 GiB   1.3 TiB  27.76  0.68   14      up          osd.46
 47    ssd    1.74660   1.00000  1.7 TiB  777 GiB   775 GiB  6.1 MiB  1.9 GiB  1012 GiB  43.43  1.07   22      up          osd.47
 48    ssd    1.74660   1.00000  1.7 TiB  814 GiB   812 GiB  7.6 MiB  1.9 GiB   974 GiB  45.54  1.12   23      up          osd.48
 49    ssd    1.74660   1.00000  1.7 TiB  392 GiB   390 GiB  7.3 MiB  1.2 GiB   1.4 TiB  21.90  0.54   11      up          osd.49
 50    ssd    1.74660   1.00000  1.7 TiB  886 GiB   883 GiB   11 MiB  2.1 GiB   903 GiB  49.51  1.22   25      up          osd.50
 67    ssd    6.98630   1.00000  7.0 TiB  2.6 TiB   2.6 TiB   22 MiB  6.6 GiB   4.4 TiB  36.63  0.90   74      up          osd.67
-11          33.18504         -   33 TiB   13 TiB    13 TiB  345 MiB   35 GiB    20 TiB  39.00  0.96    -              host pve05
 51    ssd    3.49309   1.00000  3.5 TiB  1.6 TiB   1.6 TiB   13 MiB  3.8 GiB   1.9 TiB  45.59  1.12   46      up          osd.51
 52    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB   1.1 TiB  8.2 MiB  2.7 GiB   2.4 TiB  31.53  0.77   32      up          osd.52
 53    ssd    1.74660   1.00000  1.7 TiB  813 GiB   811 GiB  5.4 MiB  2.5 GiB   975 GiB  45.47  1.12   23      up          osd.53
 54    ssd    1.74660   1.00000  1.7 TiB  595 GiB   594 GiB  7.1 MiB  1.7 GiB   1.2 TiB  33.30  0.82   17      up          osd.54
 55    ssd    1.74660   1.00000  1.7 TiB  740 GiB   738 GiB  9.3 MiB  1.8 GiB   1.0 TiB  41.37  1.02   21      up          osd.55
 56    ssd    1.74660   1.00000  1.7 TiB  495 GiB   493 GiB  4.6 MiB  1.5 GiB   1.3 TiB  27.67  0.68   14      up          osd.56
 57    ssd    1.74660   1.00000  1.7 TiB  779 GiB   777 GiB  5.5 MiB  1.9 GiB  1009 GiB  43.57  1.07   22      up          osd.57
 58    ssd    1.74660   1.00000  1.7 TiB  921 GiB   919 GiB   11 MiB  2.2 GiB   867 GiB  51.52  1.27   26      up          osd.58
 59    ssd    1.74660   1.00000  1.7 TiB  674 GiB   672 GiB  8.6 MiB  1.8 GiB   1.1 TiB  37.70  0.93   19      up          osd.59
 60    ssd    1.74660   1.00000  1.7 TiB  710 GiB   708 GiB  9.3 MiB  2.0 GiB   1.1 TiB  39.71  0.98   20      up          osd.60
 61    ssd    1.74660   1.00000  1.7 TiB  930 GiB   927 GiB  6.5 MiB  3.1 GiB   859 GiB  52.00  1.28   26      up          osd.61
 62    ssd    1.74660   1.00000  1.7 TiB  635 GiB   633 GiB   11 MiB  1.7 GiB   1.1 TiB  35.52  0.87   18      up          osd.62
 63    ssd    1.74660   1.00000  1.7 TiB  885 GiB   883 GiB  9.9 MiB  2.0 GiB   903 GiB  49.49  1.22   25      up          osd.63
 68    ssd    6.98630   1.00000  7.0 TiB  2.3 TiB   2.3 TiB  235 MiB  6.1 GiB   4.7 TiB  32.37  0.79   66      up          osd.68
                          TOTAL  131 TiB   53 TiB    53 TiB  1.0 GiB  149 GiB    77 TiB  40.71                                   
MIN/MAX VAR: 0.54/1.87  STDDEV: 10.08

Please excuse the pve02 missing data at the moment, as we are experiencing a fault with this node since yesterday and we are waiting for a replacement from our vendor. But I remember that the numbers on it were pretty much the same as on every other node.

Code: 
root@pve01 ~ # pveceph pool ls --noborder
Name                  Size Min Size PG Num min. PG Num Optimal PG Num PG Autoscale Mode PG Autoscale Target Size PG Autoscale Target Ratio Crush Rule Name               %-Used Used
ceph-vms01               3        2    512                       1024 off                                                                  replicated_rule     0.63558954000473 58278231149830
device_health_metrics    3        2      1           1              1 on                                                                   replicated_rule 1.65291039593285e-05 552302422
 
Hmm… The OSD sizes are all over the place, that does make it a lot harder to understand and do estimate calculations of what you can expect. One thing though: to give the autoscaler a hint of what you expect in the future, it is always a good idea to set the target_ratio or target_size.

Since you only have one pool in use (ignore the device_health_metrics), you can just set the target_ratio to any value. It is a weight and if there is only one pool with it configured, any value will result in it being weighted to 100%.

I would expect the autoscaler to determine a higher pg_num for the pool to be better. More PGs means each PG will hold less data -> better distribution across the OSDs.
And set the autoscale-mode to at least warn. Then you can manually change the pg_num for the pool in the GUI.

Another thing you can check for is how the balancer is doing. The balancer is there to improve the distribution after the CRUSH algorithm.
ceph balancer status
 
Thanks for the hints.
I've set the autoscaler mode to warn at the moment and the ratio to 1

Code: 
root@pve01 ~ # pveceph pool ls --noborder
Name                  Size Min Size PG Num min. PG Num Optimal PG Num PG Autoscale Mode PG Autoscale Target Size PG Autoscale Target Ratio Crush Rule Name              %-Used Used
ceph-vms01               3        2    512                       2048 warn                                                               1 replicated_rule   0.635470151901245 58277692450704
device_health_metrics    3        2      1           1              1 on                                                                   replicated_rule 1.6390360542573e-05 547943683
It now suggests 2048 PGs.

Regarding the number of PGs, in the past there were fewer and we increased them based on the autoscaler's suggestion (at that time - without the 7.68TB drives), but there were no improvements in terms of usable space.
Since this operation is extremely heavy on the cluster, causing delays on the VMs, we avoid changing them if there is not a valid reason to, that's why we disabled the autoscaler.

Here's the output of ceph balancer status

Code: 
root@pve01 ~ # ceph balancer status
{
    "active": false,
    "last_optimize_duration": "",
    "last_optimize_started": "",
    "mode": "none",
    "optimize_result": "",
    "plans": []
}

So, should we increase the PGs to 2048 as the autoscaler suggests now?
Should we do this operation now that PVE02 is down or should we wait for the cluster to be healthy again before attempting any changes to it?
 
aaron said:
Hmm… The OSD sizes are all over the place, that does make it a lot harder to understand and do estimate calculations of what you can expect.
Click to expand...
Would re-arranging the available disks per node make any difference in either easier calculations or actual usable space?
 
Which versions are you running? That the balancer is present but not enabled hints at it being a somewhat older version.

Cha0s said:
Regarding the number of PGs, in the past there were fewer and we increased them based on the autoscaler's suggestion (at that time - without the 7.68TB drives), but there were no improvements in terms of usable space.
Click to expand...
With no target_ratio or target_size the autoscaler can only use the current used space of the pool for its consideration. Telling the autoscaler what the expected end result is, will reduce later changes to the PG num as it should only occur if you make changes to the cluster, like adding more OSDs.

Cha0s said:
Would re-arranging the available disks per node make any difference in either easier calculations or actual usable space?
Click to expand...
In the end, the nodes should be as similar as possible. You can see in the ceph osd df tree output that the nodes also have a weight. The weight of each OSD or node correlates with the raw storage space available and is used by the CRUSH algorithm to decide where to place the data.

Nodes or OSDs with a height weight will get more data. That also increases the number of accesses to the resource. That is why keeping everything as evenly as possible will also distribute the load more evenly, avoiding bottlenecks.

From what I can see, it is quite even already. Changing the disks around is something I would avoid for now. First of, you don't want nodes with considerably less raw storage capacity, and it will cause a lot of additional traffic.

Setting the pg_num higher is rather what you should try first. Also, if the resulting rebalance operation has an impact on production, you can try to limit the speed.
 
This is ceph octopus (15.2.15). We haven't upgraded yet to a newer version but it's on our todo list.


Ok, we will increase the pg_num first. Can we try that now (while pve02 is down) or should we wait until its fixed first?
I also enabled the balancer but from what I read on the documentation it won't do anything until all OSDs are back online.

If it helps this is the score that the balancer reports
Code: 
root@pve01 ~ # ceph balancer eval
current cluster score 0.060641 (lower is better)


Thank you for the valuable information.
 
So, we brought pve02 back online, and changed the pg_num to 2048.
After the rebalancing completed, we gained tons of usable storage!

From 28.1TiB it went to 44.24TiB, which is a huge gain of 16.14TiB!
But still, there are almost 10TiB that are not accounted for.

ceph reports the following raw storage numbers:
Used: 53.21 TiB of 162.41 TiB

But the pool shows
Used: 17.67 TiB of 44.24 TiB

So: 17.67 TiB x 3 replicas = 53.01TiB which is almost matched with the raw number of 53.21TiB
But the usable space doesn't add up. 44.24TiB x 3replicas = 132,72 TiB of raw storage, which is 29.69TiB less than the actual available raw storage, ie: 29.69 TiB / 3 replicas = 9.89 TiB less usable space.

In terms of percentages, ceph reports a 33% raw disk usage while the pool reports a 39.9% usage.

Any ideas why is there so much space still missing?


Here's the current status of the cluster:

Code: 
root@pve01 ~ # ceph osd df tree
ID   CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP     META      AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME     
 -1         162.40396         -  162 TiB   53 TiB   53 TiB  1.2 GiB   186 GiB  109 TiB  32.76  1.00    -          root default 
 -3          31.86797         -   32 TiB   11 TiB   10 TiB  298 MiB    41 GiB   21 TiB  32.95  1.01    -              host pve01
  0    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  162 MiB   4.3 GiB  2.3 TiB  34.99  1.07  141      up          osd.0
  1    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  7.0 MiB   3.6 GiB  2.3 TiB  33.65  1.03  136      up          osd.1
  2    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB  1.1 TiB   22 MiB   4.3 GiB  2.4 TiB  31.51  0.96  126      up          osd.2
  3    ssd    3.49309   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   17 MiB   4.9 GiB  2.3 TiB  35.25  1.08  142      up          osd.3
 12    ssd    1.74599   1.00000  1.7 TiB  692 GiB  689 GiB  8.0 MiB   2.7 GiB  1.1 TiB  38.69  1.18   78      up          osd.12
 13    ssd    1.74599   1.00000  1.7 TiB  545 GiB  542 GiB  7.1 MiB   2.5 GiB  1.2 TiB  30.47  0.93   62      up          osd.13
 14    ssd    1.74599   1.00000  1.7 TiB  479 GiB  477 GiB  7.0 MiB   2.1 GiB  1.3 TiB  26.80  0.82   54      up          osd.14
 16    ssd    1.74599   1.00000  1.7 TiB  606 GiB  604 GiB   11 MiB   2.5 GiB  1.2 TiB  33.89  1.03   68      up          osd.16
 26    ssd    0.72699   1.00000  745 GiB  265 GiB  263 GiB   12 MiB   1.6 GiB  480 GiB  35.50  1.08   30      up          osd.26
 27    ssd    0.72699   1.00000  745 GiB  233 GiB  231 GiB  6.2 MiB   1.8 GiB  512 GiB  31.24  0.95   26      up          osd.27
 28    ssd    0.72699   1.00000  745 GiB  195 GiB  193 GiB  8.2 MiB   1.7 GiB  550 GiB  26.18  0.80   22      up          osd.28
 38    ssd    1.74660   1.00000  1.7 TiB  678 GiB  676 GiB  8.9 MiB   1.9 GiB  1.1 TiB  37.93  1.16   77      up          osd.38
 64    ssd    6.98630   1.00000  7.0 TiB  2.2 TiB  2.2 TiB   22 MiB   7.5 GiB  4.8 TiB  31.03  0.95  251      up          osd.64
 -5          31.86577         -   32 TiB   11 TiB   11 TiB  371 MiB    38 GiB   21 TiB  33.55  1.02    -              host pve02
  4    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   13 MiB   3.7 GiB  2.3 TiB  34.29  1.05  139      up          osd.4
  5    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.1 TiB   13 MiB   3.7 GiB  2.3 TiB  32.97  1.01  133      up          osd.5
  6    ssd    3.49199   1.00000  3.5 TiB  1.3 TiB  1.3 TiB   13 MiB   4.0 GiB  2.2 TiB  36.17  1.10  146      up          osd.6
  7    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   11 MiB   4.1 GiB  2.3 TiB  33.75  1.03  136      up          osd.7
 17    ssd    1.74599   1.00000  1.7 TiB  526 GiB  523 GiB  9.0 MiB   2.3 GiB  1.2 TiB  29.41  0.90   59      up          osd.17
 18    ssd    1.74660   1.00000  1.7 TiB  590 GiB  588 GiB   10 MiB   1.9 GiB  1.2 TiB  33.00  1.01   67      up          osd.18
 19    ssd    1.74599   1.00000  1.7 TiB  548 GiB  546 GiB  6.4 MiB   2.2 GiB  1.2 TiB  30.64  0.94   63      up          osd.19
 20    ssd    1.74599   1.00000  1.7 TiB  613 GiB  610 GiB  7.5 MiB   2.7 GiB  1.1 TiB  34.28  1.05   68      up          osd.20
 21    ssd    1.74599   1.00000  1.7 TiB  560 GiB  557 GiB  248 MiB   2.5 GiB  1.2 TiB  31.32  0.96   64      up          osd.21
 29    ssd    0.72699   1.00000  745 GiB  181 GiB  180 GiB  5.1 MiB  1019 MiB  564 GiB  24.25  0.74   20      up          osd.29
 30    ssd    0.72699   1.00000  745 GiB  281 GiB  280 GiB  4.5 MiB   1.1 GiB  464 GiB  37.76  1.15   31      up          osd.30
 31    ssd    0.72699   1.00000  745 GiB  245 GiB  243 GiB  6.7 MiB   1.7 GiB  500 GiB  32.87  1.00   27      up          osd.31
 65    ssd    6.98630   1.00000  7.0 TiB  2.4 TiB  2.4 TiB   23 MiB   7.3 GiB  4.5 TiB  34.95  1.07  283      up          osd.65
 -7          32.30014         -   32 TiB   11 TiB   11 TiB  134 MiB    39 GiB   21 TiB  33.80  1.03    -              host pve03
  8    ssd    3.49199   1.00000  3.5 TiB  1.0 TiB  1.0 TiB  9.4 MiB   3.7 GiB  2.5 TiB  29.28  0.89  118      up          osd.8
  9    ssd    3.49199   1.00000  3.5 TiB  1.1 TiB  1.1 TiB   13 MiB   3.8 GiB  2.4 TiB  31.79  0.97  127      up          osd.9
 10    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   13 MiB   3.5 GiB  2.3 TiB  33.40  1.02  135      up          osd.10
 11    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   15 MiB   3.4 GiB  2.3 TiB  33.31  1.02  135      up          osd.11
 22    ssd    1.74599   1.00000  1.7 TiB  555 GiB  552 GiB  3.8 MiB   2.4 GiB  1.2 TiB  31.03  0.95   62      up          osd.22
 23    ssd    1.74599   1.00000  1.7 TiB  499 GiB  497 GiB   11 MiB   2.2 GiB  1.3 TiB  27.93  0.85   56      up          osd.23
 24    ssd    1.74599   1.00000  1.7 TiB  749 GiB  746 GiB   10 MiB   2.7 GiB  1.0 TiB  41.89  1.28   84      up          osd.24
 25    ssd    1.74599   1.00000  1.7 TiB  547 GiB  545 GiB  6.4 MiB   2.1 GiB  1.2 TiB  30.61  0.93   62      up          osd.25
 32    ssd    0.72699   1.00000  745 GiB  304 GiB  303 GiB  5.1 MiB   1.4 GiB  441 GiB  40.85  1.25   34      up          osd.32
 33    ssd    0.72699   1.00000  745 GiB  306 GiB  305 GiB  4.8 MiB   1.1 GiB  439 GiB  41.07  1.25   34      up          osd.33
 34    ssd    0.72699   1.00000  745 GiB  330 GiB  328 GiB  4.2 MiB   1.2 GiB  415 GiB  44.23  1.35   37      up          osd.34
 35    ssd    0.72699   1.00000  745 GiB  342 GiB  341 GiB  3.8 MiB   1.2 GiB  403 GiB  45.93  1.40   38      up          osd.35
 36    ssd    0.72699   1.00000  745 GiB  321 GiB  319 GiB  3.9 MiB   1.1 GiB  424 GiB  43.03  1.31   36      up          osd.36
 37    ssd    0.72699   1.00000  745 GiB  224 GiB  223 GiB  4.1 MiB  1020 MiB  521 GiB  30.01  0.92   25      up          osd.37
 66    ssd    6.98630   1.00000  7.0 TiB  2.4 TiB  2.4 TiB   27 MiB   8.2 GiB  4.6 TiB  34.01  1.04  275      up          osd.66
 -9          33.18504         -   33 TiB   11 TiB   11 TiB  134 MiB    34 GiB   23 TiB  32.06  0.98    -              host pve04
 15    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB  1.1 TiB   11 MiB   3.7 GiB  2.4 TiB  32.39  0.99  130      up          osd.15
 39    ssd    3.49309   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   14 MiB   3.3 GiB  2.3 TiB  33.98  1.04  137      up          osd.39
 40    ssd    1.74660   1.00000  1.7 TiB  548 GiB  547 GiB  7.2 MiB   1.7 GiB  1.2 TiB  30.66  0.94   62      up          osd.40
 41    ssd    1.74660   1.00000  1.7 TiB  520 GiB  519 GiB  8.7 MiB   1.6 GiB  1.2 TiB  29.10  0.89   59      up          osd.41
 42    ssd    1.74660   1.00000  1.7 TiB  533 GiB  530 GiB  5.0 MiB   2.2 GiB  1.2 TiB  29.78  0.91   60      up          osd.42
 43    ssd    1.74660   1.00000  1.7 TiB  565 GiB  563 GiB  5.7 MiB   1.7 GiB  1.2 TiB  31.57  0.96   64      up          osd.43
 44    ssd    1.74660   1.00000  1.7 TiB  640 GiB  639 GiB   11 MiB   1.8 GiB  1.1 TiB  35.81  1.09   72      up          osd.44
 45    ssd    1.74660   1.00000  1.7 TiB  678 GiB  676 GiB  7.4 MiB   1.8 GiB  1.1 TiB  37.89  1.16   77      up          osd.45
 46    ssd    1.74660   1.00000  1.7 TiB  571 GiB  569 GiB  8.7 MiB   2.0 GiB  1.2 TiB  31.90  0.97   65      up          osd.46
 47    ssd    1.74660   1.00000  1.7 TiB  497 GiB  495 GiB   11 MiB   1.6 GiB  1.3 TiB  27.76  0.85   56      up          osd.47
 48    ssd    1.74660   1.00000  1.7 TiB  660 GiB  658 GiB  8.0 MiB   1.7 GiB  1.1 TiB  36.89  1.13   75      up          osd.48
 49    ssd    1.74660   1.00000  1.7 TiB  527 GiB  525 GiB   10 MiB   1.5 GiB  1.2 TiB  29.47  0.90   60      up          osd.49
 50    ssd    1.74660   1.00000  1.7 TiB  540 GiB  539 GiB  8.6 MiB   1.7 GiB  1.2 TiB  30.22  0.92   61      up          osd.50
 67    ssd    6.98630   1.00000  7.0 TiB  2.2 TiB  2.2 TiB   18 MiB   7.6 GiB  4.8 TiB  31.34  0.96  254      up          osd.67
-11          33.18504         -   33 TiB   10 TiB   10 TiB  306 MiB    33 GiB   23 TiB  31.49  0.96    -              host pve05
 51    ssd    3.49309   1.00000  3.5 TiB  1.3 TiB  1.3 TiB   20 MiB   4.0 GiB  2.2 TiB  36.77  1.12  148      up          osd.51
 52    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB  1.0 TiB  8.6 MiB   2.6 GiB  2.4 TiB  30.09  0.92  121      up          osd.52
 53    ssd    1.74660   1.00000  1.7 TiB  563 GiB  561 GiB   13 MiB   1.6 GiB  1.2 TiB  31.46  0.96   63      up          osd.53
 54    ssd    1.74660   1.00000  1.7 TiB  439 GiB  437 GiB   11 MiB   1.5 GiB  1.3 TiB  24.52  0.75   50      up          osd.54
 55    ssd    1.74660   1.00000  1.7 TiB  471 GiB  469 GiB  6.4 MiB   1.5 GiB  1.3 TiB  26.32  0.80   53      up          osd.55
 56    ssd    1.74660   1.00000  1.7 TiB  458 GiB  457 GiB  8.0 MiB   1.4 GiB  1.3 TiB  25.61  0.78   52      up          osd.56
 57    ssd    1.74660   1.00000  1.7 TiB  557 GiB  556 GiB  6.0 MiB   1.8 GiB  1.2 TiB  31.16  0.95   63      up          osd.57
 58    ssd    1.74660   1.00000  1.7 TiB  683 GiB  681 GiB  6.5 MiB   2.0 GiB  1.1 TiB  38.16  1.17   77      up          osd.58
 59    ssd    1.74660   1.00000  1.7 TiB  546 GiB  544 GiB  5.9 MiB   1.6 GiB  1.2 TiB  30.52  0.93   61      up          osd.59
 60    ssd    1.74660   1.00000  1.7 TiB  530 GiB  528 GiB  3.1 MiB   2.0 GiB  1.2 TiB  29.64  0.90   60      up          osd.60
 61    ssd    1.74660   1.00000  1.7 TiB  753 GiB  751 GiB   11 MiB   2.0 GiB  1.0 TiB  42.12  1.29   85      up          osd.61
 62    ssd    1.74660   1.00000  1.7 TiB  583 GiB  581 GiB  8.5 MiB   1.5 GiB  1.2 TiB  32.59  0.99   66      up          osd.62
 63    ssd    1.74660   1.00000  1.7 TiB  520 GiB  517 GiB  4.3 MiB   2.3 GiB  1.2 TiB  29.05  0.89   58      up          osd.63
 68    ssd    6.98630   1.00000  7.0 TiB  2.2 TiB  2.1 TiB  192 MiB   7.2 GiB  4.8 TiB  30.84  0.94  251      up          osd.68
                          TOTAL  162 TiB   53 TiB   53 TiB  1.2 GiB   186 GiB  109 TiB  32.76                                   
MIN/MAX VAR: 0.74/1.40  STDDEV: 4.60

Code: 
root@pve01 ~ # pveceph pool ls --noborder
Name                  Size Min Size PG Num min. PG Num Optimal PG Num PG Autoscale Mode PG Autoscale Target Size PG Autoscale Target Ratio Crush Rule Name               %-Used Used
ceph-vms01               3        2   2048                       2048 warn                                                               1 replicated_rule    0.399345248937607 58289677459296
device_health_metrics    3        2      1           1              1 on                                                                   replicated_rule 8.65226411406184e-06 758580257



I also enabled the balancer and set it to upmap mode. To test if it actually works, I tempoarily reduced target_max_misplaced_ratio to 0.02, and the cluster actually started rebalancing itself after manually executing the balancer plan I created. I then reverted target_max_misplaced_ratio back to the default 0.05.

Code: 
root@pve01 ~ # ceph balancer status
{
    "active": false,
    "last_optimize_duration": "0:00:00.013173",
    "last_optimize_started": "Wed Jan  3 22:41:32 2024",
    "mode": "upmap",
    "optimize_result": "Optimization plan created successfully",
    "plans": []
}
root@pve01 ~ # ceph balancer eval
current cluster score 0.031198 (lower is better)

So the balancer seems to be working ok and should kick in automatically when there's 5% improvment to be done.
 
AFAICT some OSDs are still quite a bit fuller than others, percentage wise. And the Ceph balancer isn't active (active: false). IIRC Octopus still needed you to enable it. On newer versions it is always on.

According to https://docs.ceph.com/en/octopus/rados/operations/balancer/ a ceph balancer on should be enough. Then give it some time and observe the usable space. I expect it will grow even more as the balancer manually moves PGs from fuller OSDs to emptier ones where the CRUSH algorithm wasn't working good enough.
 
You are correct, I was looking on the latest documentation which didn't mention about ceph balancer on
I thought that it was enabled by manually creating a plan, but obviously I was wrong.

So I run ceph balancer on and it immediatelly started moving around a few PGs and the usable space grew to 48.07TiB!

Here are the current stats:

Code: 
root@pve01 ~ # ceph balancer status
{
    "active": true,
    "last_optimize_duration": "0:00:00.009997",
    "last_optimize_started": "Thu Jan  4 18:31:49 2024",
    "mode": "upmap",
    "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect",
    "plans": []
}

Code: 
root@pve01 ~ # ceph balancer eval
current cluster score 0.015280 (lower is better)


Code: 
root@pve01 ~ # ceph osd df tree
ID   CLASS  WEIGHT     REWEIGHT  SIZE     RAW USE  DATA     OMAP     META      AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME     
 -1         162.40396         -  162 TiB   53 TiB   53 TiB  1.2 GiB   178 GiB  109 TiB  32.76  1.00    -          root default 
 -3          31.86797         -   32 TiB   11 TiB   10 TiB  286 MiB    40 GiB   21 TiB  33.02  1.01    -              host pve01
  0    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  163 MiB   4.1 GiB  2.3 TiB  34.28  1.05  138      up          osd.0
  1    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  9.3 MiB   3.6 GiB  2.3 TiB  33.67  1.03  136      up          osd.1
  2    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB  1.1 TiB   20 MiB   4.3 GiB  2.4 TiB  32.20  0.98  129      up          osd.2
  3    ssd    3.49309   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   23 MiB   4.8 GiB  2.3 TiB  33.98  1.04  137      up          osd.3
 12    ssd    1.74599   1.00000  1.7 TiB  621 GiB  618 GiB  9.6 MiB   2.3 GiB  1.1 TiB  34.71  1.06   70      up          osd.12
 13    ssd    1.74599   1.00000  1.7 TiB  554 GiB  551 GiB  4.7 MiB   2.6 GiB  1.2 TiB  30.96  0.95   63      up          osd.13
 14    ssd    1.74599   1.00000  1.7 TiB  558 GiB  556 GiB   11 MiB   2.1 GiB  1.2 TiB  31.23  0.95   63      up          osd.14
 16    ssd    1.74599   1.00000  1.7 TiB  606 GiB  604 GiB  6.5 MiB   2.3 GiB  1.2 TiB  33.91  1.04   68      up          osd.16
 26    ssd    0.72699   1.00000  745 GiB  265 GiB  263 GiB  2.3 MiB   1.6 GiB  480 GiB  35.52  1.08   30      up          osd.26
 27    ssd    0.72699   1.00000  745 GiB  232 GiB  231 GiB  3.3 MiB   1.5 GiB  513 GiB  31.19  0.95   26      up          osd.27
 28    ssd    0.72699   1.00000  745 GiB  213 GiB  212 GiB  3.0 MiB   1.5 GiB  532 GiB  28.61  0.87   24      up          osd.28
 38    ssd    1.74660   1.00000  1.7 TiB  618 GiB  616 GiB  9.9 MiB   1.9 GiB  1.1 TiB  34.54  1.05   70      up          osd.38
 64    ssd    6.98630   1.00000  7.0 TiB  2.3 TiB  2.2 TiB   22 MiB   7.6 GiB  4.7 TiB  32.30  0.99  261      up          osd.64
 -5          31.86577         -   32 TiB   11 TiB   11 TiB  338 MiB    36 GiB   21 TiB  33.12  1.01    -              host pve02
  4    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  6.7 MiB   3.2 GiB  2.3 TiB  33.82  1.03  137      up          osd.4
  5    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.1 TiB   11 MiB   3.6 GiB  2.3 TiB  32.97  1.01  133      up          osd.5
  6    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   11 MiB   3.8 GiB  2.3 TiB  33.69  1.03  136      up          osd.6
  7    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  7.9 MiB   3.6 GiB  2.3 TiB  33.73  1.03  136      up          osd.7
 17    ssd    1.74599   1.00000  1.7 TiB  567 GiB  565 GiB  4.4 MiB   2.3 GiB  1.2 TiB  31.70  0.97   64      up          osd.17
 18    ssd    1.74660   1.00000  1.7 TiB  590 GiB  589 GiB  7.6 MiB   1.6 GiB  1.2 TiB  33.01  1.01   67      up          osd.18
 19    ssd    1.74599   1.00000  1.7 TiB  548 GiB  546 GiB   10 MiB   1.9 GiB  1.2 TiB  30.63  0.94   63      up          osd.19
 20    ssd    1.74599   1.00000  1.7 TiB  614 GiB  611 GiB  8.4 MiB   2.8 GiB  1.1 TiB  34.33  1.05   68      up          osd.20
 21    ssd    1.74599   1.00000  1.7 TiB  560 GiB  558 GiB  232 MiB   2.1 GiB  1.2 TiB  31.31  0.96   64      up          osd.21
 29    ssd    0.72699   1.00000  745 GiB  223 GiB  222 GiB  5.0 MiB  1019 MiB  522 GiB  29.99  0.92   25      up          osd.29
 30    ssd    0.72699   1.00000  745 GiB  281 GiB  280 GiB  4.0 MiB  1020 MiB  464 GiB  37.75  1.15   31      up          osd.30
 31    ssd    0.72699   1.00000  745 GiB  245 GiB  243 GiB  3.8 MiB   1.5 GiB  500 GiB  32.87  1.00   27      up          osd.31
 65    ssd    6.98630   1.00000  7.0 TiB  2.3 TiB  2.3 TiB   25 MiB   7.0 GiB  4.7 TiB  33.28  1.02  269      up          osd.65
 -7          32.30014         -   32 TiB   11 TiB   11 TiB  133 MiB    38 GiB   22 TiB  33.35  1.02    -              host pve03
  8    ssd    3.49199   1.00000  3.5 TiB  1.1 TiB  1.1 TiB  9.2 MiB   3.8 GiB  2.4 TiB  31.97  0.98  129      up          osd.8
  9    ssd    3.49199   1.00000  3.5 TiB  1.1 TiB  1.1 TiB   11 MiB   3.9 GiB  2.4 TiB  32.29  0.99  129      up          osd.9
 10    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  7.9 MiB   3.5 GiB  2.3 TiB  33.42  1.02  135      up          osd.10
 11    ssd    3.49199   1.00000  3.5 TiB  1.2 TiB  1.2 TiB  9.6 MiB   3.5 GiB  2.3 TiB  33.33  1.02  135      up          osd.11
 22    ssd    1.74599   1.00000  1.7 TiB  564 GiB  561 GiB   11 MiB   2.3 GiB  1.2 TiB  31.52  0.96   63      up          osd.22
 23    ssd    1.74599   1.00000  1.7 TiB  561 GiB  558 GiB   13 MiB   2.3 GiB  1.2 TiB  31.35  0.96   63      up          osd.23
 24    ssd    1.74599   1.00000  1.7 TiB  626 GiB  623 GiB  8.0 MiB   2.5 GiB  1.1 TiB  35.00  1.07   70      up          osd.24
 25    ssd    1.74599   1.00000  1.7 TiB  557 GiB  555 GiB   10 MiB   1.9 GiB  1.2 TiB  31.16  0.95   63      up          osd.25
 32    ssd    0.72699   1.00000  745 GiB  286 GiB  285 GiB  2.5 MiB   1.4 GiB  459 GiB  38.41  1.17   32      up          osd.32
 33    ssd    0.72699   1.00000  745 GiB  288 GiB  287 GiB  4.2 MiB   1.2 GiB  457 GiB  38.67  1.18   32      up          osd.33
 34    ssd    0.72699   1.00000  745 GiB  286 GiB  285 GiB  5.4 MiB   1.1 GiB  459 GiB  38.42  1.17   32      up          osd.34
 35    ssd    0.72699   1.00000  745 GiB  290 GiB  288 GiB  4.6 MiB   1.2 GiB  455 GiB  38.88  1.19   32      up          osd.35
 36    ssd    0.72699   1.00000  745 GiB  286 GiB  285 GiB  5.2 MiB   1.0 GiB  459 GiB  38.38  1.17   32      up          osd.36
 37    ssd    0.72699   1.00000  745 GiB  224 GiB  223 GiB  3.3 MiB  1021 MiB  521 GiB  30.01  0.92   25      up          osd.37
 66    ssd    6.98630   1.00000  7.0 TiB  2.3 TiB  2.3 TiB   27 MiB   7.9 GiB  4.7 TiB  33.28  1.02  269      up          osd.66
 -9          33.18504         -   33 TiB   11 TiB   11 TiB  146 MiB    32 GiB   22 TiB  32.29  0.99    -              host pve04
 15    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB  1.1 TiB  9.8 MiB   3.6 GiB  2.4 TiB  32.41  0.99  130      up          osd.15
 39    ssd    3.49309   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   14 MiB   3.5 GiB  2.3 TiB  34.00  1.04  137      up          osd.39
 40    ssd    1.74660   1.00000  1.7 TiB  557 GiB  555 GiB  7.1 MiB   1.6 GiB  1.2 TiB  31.13  0.95   63      up          osd.40
 41    ssd    1.74660   1.00000  1.7 TiB  555 GiB  553 GiB   13 MiB   1.6 GiB  1.2 TiB  31.02  0.95   63      up          osd.41
 42    ssd    1.74660   1.00000  1.7 TiB  551 GiB  550 GiB  8.1 MiB   1.6 GiB  1.2 TiB  30.83  0.94   62      up          osd.42
 43    ssd    1.74660   1.00000  1.7 TiB  564 GiB  562 GiB  6.2 MiB   1.5 GiB  1.2 TiB  31.52  0.96   64      up          osd.43
 44    ssd    1.74660   1.00000  1.7 TiB  623 GiB  621 GiB  9.9 MiB   1.6 GiB  1.1 TiB  34.82  1.06   70      up          osd.44
 45    ssd    1.74660   1.00000  1.7 TiB  617 GiB  615 GiB   14 MiB   1.8 GiB  1.1 TiB  34.48  1.05   70      up          osd.45
 46    ssd    1.74660   1.00000  1.7 TiB  571 GiB  569 GiB  7.2 MiB   1.6 GiB  1.2 TiB  31.90  0.97   65      up          osd.46
 47    ssd    1.74660   1.00000  1.7 TiB  549 GiB  548 GiB  6.3 MiB   1.4 GiB  1.2 TiB  30.71  0.94   62      up          osd.47
 48    ssd    1.74660   1.00000  1.7 TiB  617 GiB  615 GiB   11 MiB   1.7 GiB  1.1 TiB  34.47  1.05   70      up          osd.48
 49    ssd    1.74660   1.00000  1.7 TiB  544 GiB  543 GiB  8.0 MiB   1.5 GiB  1.2 TiB  30.42  0.93   62      up          osd.49
 50    ssd    1.74660   1.00000  1.7 TiB  549 GiB  547 GiB  8.8 MiB   1.6 GiB  1.2 TiB  30.67  0.94   62      up          osd.50
 67    ssd    6.98630   1.00000  7.0 TiB  2.2 TiB  2.2 TiB   23 MiB   7.6 GiB  4.7 TiB  32.19  0.98  261      up          osd.67
-11          33.18504         -   33 TiB   11 TiB   11 TiB  319 MiB    32 GiB   23 TiB  32.06  0.98    -              host pve05
 51    ssd    3.49309   1.00000  3.5 TiB  1.2 TiB  1.2 TiB   20 MiB   3.8 GiB  2.3 TiB  34.04  1.04  137      up          osd.51
 52    ssd    3.49309   1.00000  3.5 TiB  1.1 TiB  1.1 TiB  8.6 MiB   2.6 GiB  2.4 TiB  31.80  0.97  128      up          osd.52
 53    ssd    1.74660   1.00000  1.7 TiB  563 GiB  561 GiB   10 MiB   1.5 GiB  1.2 TiB  31.46  0.96   63      up          osd.53
 54    ssd    1.74660   1.00000  1.7 TiB  544 GiB  543 GiB   10 MiB   1.4 GiB  1.2 TiB  30.42  0.93   62      up          osd.54
 55    ssd    1.74660   1.00000  1.7 TiB  553 GiB  551 GiB  6.9 MiB   1.8 GiB  1.2 TiB  30.93  0.94   62      up          osd.55
 56    ssd    1.74660   1.00000  1.7 TiB  543 GiB  542 GiB  6.2 MiB   1.7 GiB  1.2 TiB  30.39  0.93   62      up          osd.56
 57    ssd    1.74660   1.00000  1.7 TiB  557 GiB  556 GiB  8.8 MiB   1.7 GiB  1.2 TiB  31.17  0.95   63      up          osd.57
 58    ssd    1.74660   1.00000  1.7 TiB  623 GiB  621 GiB   12 MiB   1.8 GiB  1.1 TiB  34.85  1.06   70      up          osd.58
 59    ssd    1.74660   1.00000  1.7 TiB  554 GiB  553 GiB  7.2 MiB   1.6 GiB  1.2 TiB  30.99  0.95   62      up          osd.59
 60    ssd    1.74660   1.00000  1.7 TiB  548 GiB  546 GiB  7.2 MiB   1.6 GiB  1.2 TiB  30.64  0.94   62      up          osd.60
 61    ssd    1.74660   1.00000  1.7 TiB  621 GiB  619 GiB   11 MiB   1.8 GiB  1.1 TiB  34.71  1.06   70      up          osd.61
 62    ssd    1.74660   1.00000  1.7 TiB  583 GiB  581 GiB  6.8 MiB   1.6 GiB  1.2 TiB  32.59  0.99   66      up          osd.62
 63    ssd    1.74660   1.00000  1.7 TiB  554 GiB  552 GiB  9.0 MiB   1.6 GiB  1.2 TiB  30.98  0.95   62      up          osd.63
 68    ssd    6.98630   1.00000  7.0 TiB  2.2 TiB  2.2 TiB  194 MiB   6.9 GiB  4.7 TiB  32.07  0.98  261      up          osd.68
                          TOTAL  162 TiB   53 TiB   53 TiB  1.2 GiB   178 GiB  109 TiB  32.76                                   
MIN/MAX VAR: 0.87/1.19  STDDEV: 2.28

So now the fullest OSD is at 39% and the emptiest at 29%.

I don't mean to be ungrateful or anything, but there's still ~6TiB of usable space that are missing.
Any ideas?
 
The balancer will be happy once the VARiance is below 5%. Not all OSDs are there yet and you will never get everything as there will not be a 100% perfect balance. Also keep in mind that you should never fill up the cluster too much. Otherwise Ceph might not have enough space to recover from failures (full node, OSDs) and running out of space is one of the few very painful situations to get out from.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back