Hi,
we have run into a bit of an understanding issue with our ceph cluster in proxmox.
We recently added three new OSD's to our ceph cluster (24 to 27 OSD's) each OSD is 3,84TiB in size.
after adding the three new OSD's we expected the max_available space for our pools to increase, this was however not the case.
The cluster available space increase by 10TiB but the max_available space for the pools did not.
our current cluster distribution is as follows:
Our current theory is that the way the OSD's are distributed among the hosts is the reason for the max_available space not increasing.
Could anyone clarify/confirm if this theory is correct or if the issue is somewhere else?
we have run into a bit of an understanding issue with our ceph cluster in proxmox.
We recently added three new OSD's to our ceph cluster (24 to 27 OSD's) each OSD is 3,84TiB in size.
after adding the three new OSD's we expected the max_available space for our pools to increase, this was however not the case.
The cluster available space increase by 10TiB but the max_available space for the pools did not.
our current cluster distribution is as follows:
Code:
root@prx005:~# ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 94.31337 root default
-3 27.94470 host prx005
0 ssd 3.49309 osd.0 up 1.00000 1.00000
1 ssd 3.49309 osd.1 up 1.00000 1.00000
2 ssd 3.49309 osd.2 up 1.00000 1.00000
11 ssd 3.49309 osd.11 up 1.00000 1.00000
12 ssd 3.49309 osd.12 up 1.00000 1.00000
15 ssd 3.49309 osd.15 up 1.00000 1.00000
18 ssd 3.49309 osd.18 up 1.00000 1.00000
21 ssd 3.49309 osd.21 up 1.00000 1.00000
-5 27.94470 host prx006
3 ssd 3.49309 osd.3 up 1.00000 1.00000
4 ssd 3.49309 osd.4 up 1.00000 1.00000
5 ssd 3.49309 osd.5 up 1.00000 1.00000
9 ssd 3.49309 osd.9 up 1.00000 1.00000
13 ssd 3.49309 osd.13 up 1.00000 1.00000
16 ssd 3.49309 osd.16 up 1.00000 1.00000
19 ssd 3.49309 osd.19 up 1.00000 1.00000
22 ssd 3.49309 osd.22 up 1.00000 1.00000
-7 27.94470 host prx007
6 ssd 3.49309 osd.6 up 1.00000 1.00000
7 ssd 3.49309 osd.7 up 1.00000 1.00000
8 ssd 3.49309 osd.8 up 1.00000 1.00000
10 ssd 3.49309 osd.10 up 1.00000 1.00000
14 ssd 3.49309 osd.14 up 1.00000 1.00000
17 ssd 3.49309 osd.17 up 1.00000 1.00000
20 ssd 3.49309 osd.20 up 1.00000 1.00000
23 ssd 3.49309 osd.23 up 1.00000 1.00000
-9 10.47926 host prx008
24 ssd 3.49309 osd.24 up 1.00000 1.00000
25 ssd 3.49309 osd.25 up 1.00000 1.00000
26 ssd 3.49309 osd.26 up 1.00000 1.00000
Code:
root@prx005:~# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 94 TiB 42 TiB 53 TiB 53 TiB 55.70
TOTAL 94 TiB 42 TiB 53 TiB 53 TiB 55.70
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 1 4 48 MiB 30 143 MiB 0 8.5 TiB
ceph-ssd 4 1024 20 TiB 5.39M 52 TiB 67.16 8.5 TiB
ceph-tmpl 9 32 19 B 3 12 KiB 0 8.5 TiB
Code:
root@prx005:~# ceph osd df
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS
0 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 415 KiB 8.7 GiB 1.6 TiB 53.42 0.96 115 up
1 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 438 KiB 8.8 GiB 1.6 TiB 55.40 0.99 121 up
2 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 442 KiB 8.9 GiB 1.6 TiB 53.54 0.96 111 up
11 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 430 KiB 8.6 GiB 1.6 TiB 54.90 0.99 115 up
12 ssd 3.49309 1.00000 3.5 TiB 2.0 TiB 2.0 TiB 445 KiB 8.7 GiB 1.5 TiB 56.33 1.01 122 up
15 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 442 KiB 8.9 GiB 1.6 TiB 53.42 0.96 112 up
18 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 426 KiB 8.7 GiB 1.6 TiB 54.97 0.99 113 up
21 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 411 KiB 8.8 GiB 1.6 TiB 53.52 0.96 110 up
3 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 434 KiB 8.7 GiB 1.6 TiB 54.51 0.98 114 up
4 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 434 KiB 8.7 GiB 1.6 TiB 53.43 0.96 111 up
5 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 407 KiB 8.8 GiB 1.6 TiB 53.48 0.96 115 up
9 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 426 KiB 8.4 GiB 1.6 TiB 53.47 0.96 110 up
13 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 418 KiB 8.4 GiB 1.6 TiB 53.38 0.96 112 up
16 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 555 KiB 8.9 GiB 1.6 TiB 53.48 0.96 113 up
19 ssd 3.49309 1.00000 3.5 TiB 2.0 TiB 2.0 TiB 469 KiB 9.0 GiB 1.5 TiB 56.94 1.02 123 up
22 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 434 KiB 8.6 GiB 1.6 TiB 55.39 0.99 115 up
6 ssd 3.49309 1.00000 3.5 TiB 2.0 TiB 2.0 TiB 504 KiB 9.1 GiB 1.5 TiB 57.34 1.03 121 up
7 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 589 KiB 8.9 GiB 1.6 TiB 53.46 0.96 110 up
8 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 414 KiB 8.6 GiB 1.6 TiB 53.43 0.96 116 up
10 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 558 KiB 8.8 GiB 1.6 TiB 53.55 0.96 112 up
14 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 438 KiB 8.8 GiB 1.6 TiB 53.45 0.96 114 up
17 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 418 KiB 8.7 GiB 1.6 TiB 53.35 0.96 111 up
20 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 414 KiB 8.6 GiB 1.6 TiB 53.25 0.96 115 up
23 ssd 3.49309 1.00000 3.5 TiB 1.9 TiB 1.9 TiB 469 KiB 10 GiB 1.6 TiB 53.34 0.96 112 up
24 ssd 3.49309 1.00000 3.5 TiB 2.4 TiB 2.4 TiB 682 KiB 9.4 GiB 1.1 TiB 67.89 1.22 143 up
25 ssd 3.49309 1.00000 3.5 TiB 2.4 TiB 2.4 TiB 522 KiB 9.5 GiB 1.1 TiB 67.79 1.22 148 up
26 ssd 3.49309 1.00000 3.5 TiB 2.4 TiB 2.3 TiB 95 MiB 9.3 GiB 1.1 TiB 67.41 1.21 146 up
TOTAL 94 TiB 53 TiB 52 TiB 107 MiB 240 GiB 42 TiB 55.70
MIN/MAX VAR: 0.96/1.22 STDDEV: 4.40Our current theory is that the way the OSD's are distributed among the hosts is the reason for the max_available space not increasing.
Could anyone clarify/confirm if this theory is correct or if the issue is somewhere else?
