Hi guys! I experimented with ceph behavior when 1 of the disks fails. The disk has been "hot" removed from the server. After a couple of days, the disc was inserted back. The disk status was down/out. I clicked the IN button, but the down status didn't change. The server has been rebooted, but the status has not changed. Without waiting for the rebalance process, the disk was removed without destroying LVM. The rebalancing process has now started, and how do I add the OSD again? Is it possible?
Code:
root@220pve01:~# ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 21 TiB 14 TiB 7.6 TiB 7.6 TiB 35.76
ssd 373 GiB 166 GiB 207 GiB 207 GiB 55.51
TOTAL 22 TiB 14 TiB 7.8 TiB 7.8 TiB 36.09
--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
device_health_metrics 1 512 14 MiB 26 42 MiB 0 2.8 TiB
Ceph_HDD 2 256 2.6 TiB 695.34k 7.8 TiB 47.65 2.8 TiB
Code:
root@220pve01:~# ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 21.65019 - 22 TiB 7.8 TiB 7.8 TiB 58 MiB 31 GiB 14 TiB 36.09 1.00 - root default
-3 4.73047 - 4.7 TiB 1.9 TiB 1.9 TiB 15 MiB 6.1 GiB 2.8 TiB 40.15 1.11 - host 220pve01
1 hdd 0.54599 1.00000 559 GiB 210 GiB 209 GiB 5.2 MiB 1.0 GiB 349 GiB 37.53 1.04 63 up osd.1
3 hdd 1.09160 1.00000 1.1 TiB 448 GiB 447 GiB 722 KiB 872 MiB 670 GiB 40.09 1.11 116 up osd.3
4 hdd 1.09160 1.00000 1.1 TiB 438 GiB 437 GiB 2.0 MiB 968 MiB 680 GiB 39.20 1.09 121 up osd.4
5 hdd 1.09160 1.00000 1.1 TiB 435 GiB 433 GiB 5.0 MiB 1.7 GiB 683 GiB 38.92 1.08 115 up osd.5
6 hdd 0.54579 1.00000 559 GiB 207 GiB 206 GiB 1.4 MiB 830 MiB 352 GiB 37.02 1.03 63 up osd.6
2 ssd 0.36389 1.00000 373 GiB 207 GiB 206 GiB 740 KiB 786 MiB 166 GiB 55.54 1.54 48 up osd.2
-13 6.00377 - 6.0 TiB 2.0 TiB 2.0 TiB 17 MiB 9.0 GiB 4.0 TiB 34.08 0.94 - host 220pve02
19 hdd 0.54579 1.00000 559 GiB 177 GiB 176 GiB 3.8 MiB 610 MiB 382 GiB 31.65 0.88 55 up osd.19
20 hdd 0.54579 1.00000 559 GiB 198 GiB 196 GiB 2.2 MiB 1.3 GiB 361 GiB 35.36 0.98 56 up osd.20
21 hdd 1.09160 1.00000 1.1 TiB 383 GiB 381 GiB 1.5 MiB 1.6 GiB 735 GiB 34.26 0.95 114 up osd.21
22 hdd 1.09160 1.00000 1.1 TiB 392 GiB 390 GiB 1.8 MiB 1.7 GiB 726 GiB 35.05 0.97 113 up osd.22
23 hdd 1.09160 1.00000 1.1 TiB 386 GiB 384 GiB 2.3 MiB 1.2 GiB 732 GiB 34.49 0.96 115 up osd.23
24 hdd 1.09160 1.00000 1.1 TiB 385 GiB 383 GiB 1.5 MiB 1.7 GiB 733 GiB 34.44 0.95 114 up osd.24
25 hdd 0.54579 1.00000 559 GiB 175 GiB 174 GiB 4.4 MiB 922 MiB 384 GiB 31.37 0.87 54 up osd.25
-7 5.45798 - 5.5 TiB 2.0 TiB 1.9 TiB 15 MiB 8.5 GiB 3.5 TiB 35.83 0.99 - host 2530pve01
7 hdd 1.09160 1.00000 1.1 TiB 396 GiB 394 GiB 2.0 MiB 1.7 GiB 722 GiB 35.45 0.98 113 up osd.7
8 hdd 1.09160 1.00000 1.1 TiB 404 GiB 403 GiB 2.0 MiB 1.4 GiB 714 GiB 36.14 1.00 117 up osd.8
9 hdd 1.09160 1.00000 1.1 TiB 404 GiB 403 GiB 862 KiB 1.2 GiB 714 GiB 36.14 1.00 119 up osd.9
10 hdd 1.09160 1.00000 1.1 TiB 384 GiB 383 GiB 1.7 MiB 1.6 GiB 733 GiB 34.39 0.95 115 up osd.10
11 hdd 0.54579 1.00000 559 GiB 208 GiB 206 GiB 6.8 MiB 1.3 GiB 351 GiB 37.18 1.03 62 up osd.11
12 hdd 0.54579 1.00000 559 GiB 206 GiB 205 GiB 1.8 MiB 1.3 GiB 353 GiB 36.86 1.02 60 up osd.12
-10 5.45798 - 5.5 TiB 1.9 TiB 1.9 TiB 10 MiB 7.3 GiB 3.5 TiB 35.03 0.97 - host 2530pve02
13 hdd 1.09160 1.00000 1.1 TiB 406 GiB 405 GiB 1.5 MiB 1.4 GiB 711 GiB 36.36 1.01 116 up osd.13
14 hdd 1.09160 1.00000 1.1 TiB 390 GiB 389 GiB 2.1 MiB 974 MiB 728 GiB 34.89 0.97 113 up osd.14
15 hdd 1.09160 1.00000 1.1 TiB 373 GiB 372 GiB 2.3 MiB 1.1 GiB 745 GiB 33.39 0.93 112 up osd.15
16 hdd 1.09160 1.00000 1.1 TiB 383 GiB 381 GiB 2.4 MiB 1.5 GiB 735 GiB 34.24 0.95 114 up osd.16
17 hdd 0.54579 1.00000 559 GiB 219 GiB 218 GiB 828 KiB 1.4 GiB 340 GiB 39.20 1.09 59 up osd.17
18 hdd 0.54579 1.00000 559 GiB 186 GiB 185 GiB 1.1 MiB 996 MiB 373 GiB 33.32 0.92 57 up osd.18
TOTAL 22 TiB 7.8 TiB 7.8 TiB 58 MiB 31 GiB 14 TiB 36.09
MIN/MAX VAR: 0.87/1.54 STDDEV: 4.48
Last edited: