Useable storage space?

C

cfgmgr

Member
Jul 25, 2023
54
11
13
Trying to gauge how much wiggle room I have on this cluster before new HW is required. We are already getting pretty close I think. OSD's are all 2TB.

Code: 
# pveceph pool ls
┌─────────────────┬──────┬──────────┬────────┬─────────────┬────────────────┬───────────────────┬──────────────────────────┬───────────────────────────┬─────────────────┬──────────────────────┬────────────────┐
│ Name            │ Size │ Min Size │ PG Num │ min. PG Num │ Optimal PG Num │ PG Autoscale Mode │ PG Autoscale Target Size │ PG Autoscale Target Ratio │ Crush Rule Name │               %-Used │           Used │
╞═════════════════╪══════╪══════════╪════════╪═════════════╪════════════════╪═══════════════════╪══════════════════════════╪═══════════════════════════╪═════════════════╪══════════════════════╪════════════════╡
│ .mgr            │    3 │        2 │      1 │           1 │              1 │ on                │                          │                           │ replicated_rule │  0.00012499802687671 │     1113931776 │
├─────────────────┼──────┼──────────┼────────┼─────────────┼────────────────┼───────────────────┼──────────────────────────┼───────────────────────────┼─────────────────┼──────────────────────┼────────────────┤
│ ceph-vm         │    3 │        2 │    512 │             │            512 │ on                │                          │                           │ replicated_rule │    0.753570139408112 │ 27247801425767 │
├─────────────────┼──────┼──────────┼────────┼─────────────┼────────────────┼───────────────────┼──────────────────────────┼───────────────────────────┼─────────────────┼──────────────────────┼────────────────┤
│ cephfs_data     │    3 │        2 │     32 │             │             32 │ on                │                          │                           │ replicated_rule │   0.0265619959682226 │   243138416640 │
├─────────────────┼──────┼──────────┼────────┼─────────────┼────────────────┼───────────────────┼──────────────────────────┼───────────────────────────┼─────────────────┼──────────────────────┼────────────────┤
│ cephfs_metadata │    3 │        2 │     32 │          16 │             16 │ on                │                          │                           │ replicated_rule │ 1.26695904327789e-05 │      112893583 │
└─────────────────┴──────┴──────────┴────────┴─────────────┴────────────────┴───────────────────┴──────────────────────────┴───────────────────────────┴─────────────────┴──────────────────────┴────────────────┘

Code: 
# ceph osd df tree
ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
-1         36.67854         -   37 TiB   25 TiB   25 TiB  4.3 MiB   72 GiB   12 TiB  68.36  1.00    -          root default
-3         12.22618         -   12 TiB  8.4 TiB  8.3 TiB  1.4 MiB   24 GiB  3.9 TiB  68.36  1.00    -              host hv01
 0    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  215 KiB  3.4 GiB  595 GiB  66.72  0.98   80      up          osd.0
 3    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  199 KiB  3.5 GiB  568 GiB  68.25  1.00   83      up          osd.3
 6    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  208 KiB  3.5 GiB  551 GiB  69.18  1.01   82      up          osd.6
 9    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  209 KiB  3.9 GiB  487 GiB  72.78  1.06   91      up          osd.9
15    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  214 KiB  3.0 GiB  600 GiB  66.45  0.97   84      up          osd.15
16    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  203 KiB  3.6 GiB  567 GiB  68.29  1.00   78      up          osd.16
17    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  210 KiB  3.2 GiB  592 GiB  66.87  0.98   79      up          osd.17
-5         12.22618         -   12 TiB  8.4 TiB  8.3 TiB  1.4 MiB   25 GiB  3.9 TiB  68.37  1.00    -              host hv02
 1    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  213 KiB  3.7 GiB  488 GiB  72.69  1.06   87      up          osd.1
 4    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  209 KiB  3.5 GiB  627 GiB  64.94  0.95   81      up          osd.4
 7    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  226 KiB  3.6 GiB  554 GiB  69.00  1.01   84      up          osd.7
10    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  204 KiB  3.7 GiB  502 GiB  71.92  1.05   83      up          osd.10
14    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  226 KiB  3.1 GiB  541 GiB  69.73  1.02   80      up          osd.14
18    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  194 KiB  3.5 GiB  615 GiB  65.60  0.96   81      up          osd.18
19    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  195 KiB  3.5 GiB  631 GiB  64.70  0.95   81      up          osd.19
-7         12.22618         -   12 TiB  8.4 TiB  8.3 TiB  1.4 MiB   23 GiB  3.9 TiB  68.36  1.00    -              host hv03
 2    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  217 KiB  3.4 GiB  558 GiB  68.82  1.01   81      up          osd.2
 5    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  207 KiB  3.6 GiB  485 GiB  72.91  1.07   88      up          osd.5
 8    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  218 KiB  3.8 GiB  536 GiB  70.05  1.02   84      up          osd.8
11    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  198 KiB  3.2 GiB  613 GiB  65.73  0.96   78      up          osd.11
12    ssd   1.74660   1.00000  1.7 TiB  1.1 TiB  1.1 TiB  195 KiB  3.0 GiB  621 GiB  65.26  0.95   80      up          osd.12
13    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  202 KiB  3.1 GiB  601 GiB  66.42  0.97   82      up          osd.13
20    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  223 KiB  2.9 GiB  549 GiB  69.30  1.01   84      up          osd.20
                        TOTAL   37 TiB   25 TiB   25 TiB  4.3 MiB   72 GiB   12 TiB  68.36
MIN/MAX VAR: 0.95/1.07  STDDEV: 2.58

This is running 7.4 and Ceph 17.x. I need to leave myself enough space to patch and eventually uplift to 8.x soon.. Can I safely run this up to 80% without causing issues during a patching/reboot, etc?

What useable with Ceph is something Im still learning (without putting myself into a corner).

Thanks!
 
cfgmgr said:
Trying to gauge how much wiggle room I have on this cluster before new HW is required. We are already getting pretty close I think. OSD's are all 2TB.


Code: 
# ceph osd df tree
ID  CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
-1         36.67854         -   37 TiB   25 TiB   25 TiB  4.3 MiB   72 GiB   12 TiB  68.36  1.00    -          root default
-3         12.22618         -   12 TiB  8.4 TiB  8.3 TiB  1.4 MiB   24 GiB  3.9 TiB  68.36  1.00    -              host hv01
 0    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  215 KiB  3.4 GiB  595 GiB  66.72  0.98   80      up          osd.0
 3    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  199 KiB  3.5 GiB  568 GiB  68.25  1.00   83      up          osd.3
 6    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  208 KiB  3.5 GiB  551 GiB  69.18  1.01   82      up          osd.6
 9    ssd   1.74660   1.00000  1.7 TiB  1.3 TiB  1.3 TiB  209 KiB  3.9 GiB  487 GiB  72.78  1.06   91      up          osd.9
15    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  214 KiB  3.0 GiB  600 GiB  66.45  0.97   84      up          osd.15
16    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  203 KiB  3.6 GiB  567 GiB  68.29  1.00   78      up          osd.16
17    ssd   1.74660   1.00000  1.7 TiB  1.2 TiB  1.2 TiB  210 KiB  3.2 GiB  592 GiB  66.87  0.98   79      up          osd.17

This is running 7.4 and Ceph 17.x. I need to leave myself enough space to patch and eventually uplift to 8.x soon.. Can I safely run this up to 80% without causing issues during a patching/reboot, etc?

What useable with Ceph is something Im still learning (without putting myself into a corner).

Thanks!
Click to expand...
No, because a diskfailure on a average usage of 80% would lead to a complete downtime because youll like reach 95% disk usage on a single osd, which will lead to a pool-downtime because of ceph osd full-ratio

Edit: Why? Because ceph always recovers on the same host if you loose a disk, means if you loose a disk with 80-85% usage, this data needs to be recoverd on the same host on the reaming disks. on 90% youll have the first issues, because backfilling wont work. on 95% ýoull have clusterwide downtime.

Edit2: In your case, a failure of one of the 7 disks would lead to 11.4 % more usage on the other 6 osds, which will lead to 91.4% which is already to high.
 
Last edited:
  • Like
Reactions: cfgmgr
cfgmgr said:
What useable with Ceph is something Im still learning (without putting myself into a corner).
Click to expand...
you have a ~37T raw/12T usable pool. as a rule of thumb, 80% utilized fs is full for the reasons @jsterr mentioned- it also takes a substantial hit in performance past that point. 80% of 12TiB ~ 9.6TiB. if you're anywhere near here, you need more OSDs.
 
  • Like
Reactions: cfgmgr
You must log in or register to reply here.
Top Bottom