How to re-add ceph disk

lDemoNl

Member
Oct 23, 2020
78
3
13
30
Hi guys! I experimented with ceph behavior when 1 of the disks fails. The disk has been "hot" removed from the server. After a couple of days, the disc was inserted back. The disk status was down/out. I clicked the IN button, but the down status didn't change. The server has been rebooted, but the status has not changed. Without waiting for the rebalance process, the disk was removed without destroying LVM. The rebalancing process has now started, and how do I add the OSD again? Is it possible?

Code:
root@220pve01:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd     21 TiB   14 TiB  7.6 TiB   7.6 TiB      35.76
ssd    373 GiB  166 GiB  207 GiB   207 GiB      55.51
TOTAL   22 TiB   14 TiB  7.8 TiB   7.8 TiB      36.09
 
--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
device_health_metrics   1  512   14 MiB       26   42 MiB      0    2.8 TiB
Ceph_HDD                2  256  2.6 TiB  695.34k  7.8 TiB  47.65    2.8 TiB

Code:
root@220pve01:~# ceph osd df tree
ID   CLASS  WEIGHT    REWEIGHT  SIZE     RAW USE  DATA     OMAP     META     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME           
 -1         21.65019         -   22 TiB  7.8 TiB  7.8 TiB   58 MiB   31 GiB   14 TiB  36.09  1.00    -          root default         
 -3          4.73047         -  4.7 TiB  1.9 TiB  1.9 TiB   15 MiB  6.1 GiB  2.8 TiB  40.15  1.11    -              host 220pve01
  1    hdd   0.54599   1.00000  559 GiB  210 GiB  209 GiB  5.2 MiB  1.0 GiB  349 GiB  37.53  1.04   63      up          osd.1       
  3    hdd   1.09160   1.00000  1.1 TiB  448 GiB  447 GiB  722 KiB  872 MiB  670 GiB  40.09  1.11  116      up          osd.3       
  4    hdd   1.09160   1.00000  1.1 TiB  438 GiB  437 GiB  2.0 MiB  968 MiB  680 GiB  39.20  1.09  121      up          osd.4       
  5    hdd   1.09160   1.00000  1.1 TiB  435 GiB  433 GiB  5.0 MiB  1.7 GiB  683 GiB  38.92  1.08  115      up          osd.5       
  6    hdd   0.54579   1.00000  559 GiB  207 GiB  206 GiB  1.4 MiB  830 MiB  352 GiB  37.02  1.03   63      up          osd.6       
  2    ssd   0.36389   1.00000  373 GiB  207 GiB  206 GiB  740 KiB  786 MiB  166 GiB  55.54  1.54   48      up          osd.2       
-13          6.00377         -  6.0 TiB  2.0 TiB  2.0 TiB   17 MiB  9.0 GiB  4.0 TiB  34.08  0.94    -              host 220pve02
 19    hdd   0.54579   1.00000  559 GiB  177 GiB  176 GiB  3.8 MiB  610 MiB  382 GiB  31.65  0.88   55      up          osd.19       
 20    hdd   0.54579   1.00000  559 GiB  198 GiB  196 GiB  2.2 MiB  1.3 GiB  361 GiB  35.36  0.98   56      up          osd.20       
 21    hdd   1.09160   1.00000  1.1 TiB  383 GiB  381 GiB  1.5 MiB  1.6 GiB  735 GiB  34.26  0.95  114      up          osd.21       
 22    hdd   1.09160   1.00000  1.1 TiB  392 GiB  390 GiB  1.8 MiB  1.7 GiB  726 GiB  35.05  0.97  113      up          osd.22       
 23    hdd   1.09160   1.00000  1.1 TiB  386 GiB  384 GiB  2.3 MiB  1.2 GiB  732 GiB  34.49  0.96  115      up          osd.23       
 24    hdd   1.09160   1.00000  1.1 TiB  385 GiB  383 GiB  1.5 MiB  1.7 GiB  733 GiB  34.44  0.95  114      up          osd.24       
 25    hdd   0.54579   1.00000  559 GiB  175 GiB  174 GiB  4.4 MiB  922 MiB  384 GiB  31.37  0.87   54      up          osd.25       
 -7          5.45798         -  5.5 TiB  2.0 TiB  1.9 TiB   15 MiB  8.5 GiB  3.5 TiB  35.83  0.99    -              host 2530pve01
  7    hdd   1.09160   1.00000  1.1 TiB  396 GiB  394 GiB  2.0 MiB  1.7 GiB  722 GiB  35.45  0.98  113      up          osd.7       
  8    hdd   1.09160   1.00000  1.1 TiB  404 GiB  403 GiB  2.0 MiB  1.4 GiB  714 GiB  36.14  1.00  117      up          osd.8       
  9    hdd   1.09160   1.00000  1.1 TiB  404 GiB  403 GiB  862 KiB  1.2 GiB  714 GiB  36.14  1.00  119      up          osd.9       
 10    hdd   1.09160   1.00000  1.1 TiB  384 GiB  383 GiB  1.7 MiB  1.6 GiB  733 GiB  34.39  0.95  115      up          osd.10       
 11    hdd   0.54579   1.00000  559 GiB  208 GiB  206 GiB  6.8 MiB  1.3 GiB  351 GiB  37.18  1.03   62      up          osd.11       
 12    hdd   0.54579   1.00000  559 GiB  206 GiB  205 GiB  1.8 MiB  1.3 GiB  353 GiB  36.86  1.02   60      up          osd.12       
-10          5.45798         -  5.5 TiB  1.9 TiB  1.9 TiB   10 MiB  7.3 GiB  3.5 TiB  35.03  0.97    -              host 2530pve02
 13    hdd   1.09160   1.00000  1.1 TiB  406 GiB  405 GiB  1.5 MiB  1.4 GiB  711 GiB  36.36  1.01  116      up          osd.13       
 14    hdd   1.09160   1.00000  1.1 TiB  390 GiB  389 GiB  2.1 MiB  974 MiB  728 GiB  34.89  0.97  113      up          osd.14       
 15    hdd   1.09160   1.00000  1.1 TiB  373 GiB  372 GiB  2.3 MiB  1.1 GiB  745 GiB  33.39  0.93  112      up          osd.15       
 16    hdd   1.09160   1.00000  1.1 TiB  383 GiB  381 GiB  2.4 MiB  1.5 GiB  735 GiB  34.24  0.95  114      up          osd.16       
 17    hdd   0.54579   1.00000  559 GiB  219 GiB  218 GiB  828 KiB  1.4 GiB  340 GiB  39.20  1.09   59      up          osd.17       
 18    hdd   0.54579   1.00000  559 GiB  186 GiB  185 GiB  1.1 MiB  996 MiB  373 GiB  33.32  0.92   57      up          osd.18       
                         TOTAL   22 TiB  7.8 TiB  7.8 TiB   58 MiB   31 GiB   14 TiB  36.09                                         
MIN/MAX VAR: 0.87/1.54  STDDEV: 4.48
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!