ceph stale and not recovered

C

cola16

Member
Feb 2, 2024
35
2
8
I accidentally set the FAILURE DOMAIN of CEPH to OSD.
This time, I ended up shutting down the OSD, and one of the PROXMOX nodes was shut down due to a failure.
Since then, it has been unable to recover.


Bash: 
ceph pg dump_stuck stale
PG_STAT  STATE               UP         UP_PRIMARY  ACTING     ACTING_PRIMARY
53.42    stale+active+clean    [6,8,9]           6    [6,8,9]               6
41.5f    stale+active+clean  [12,11,9]          12  [12,11,9]              12
41.19    stale+active+clean   [8,11,6]           8   [8,11,6]               8
41.10    stale+active+clean   [12,9,8]          12   [12,9,8]              12
53.c     stale+active+clean   [6,12,8]           6   [6,12,8]               6
41.11    stale+active+clean  [11,8,12]          11  [11,8,12]              11
53.d     stale+active+clean   [11,9,8]          11   [11,9,8]              11
53.12    stale+active+clean   [6,9,11]           6   [6,9,11]               6
41.a     stale+active+clean   [6,12,8]           6   [6,12,8]               6
41.b     stale+active+clean  [8,12,11]           8  [8,12,11]               8
53.14    stale+active+clean  [6,12,11]           6  [6,12,11]               6
53.15    stale+active+clean   [8,11,6]           8   [8,11,6]               8
41.6     stale+active+clean   [9,8,11]           9   [9,8,11]               9
41.4     stale+active+clean   [8,9,12]           8   [8,9,12]               8
41.5     stale+active+clean    [6,9,8]           6    [6,9,8]               6
53.19    stale+active+clean  [8,12,11]           8  [8,12,11]               8
41.2     stale+active+clean  [9,12,11]           9  [9,12,11]               9
53.1e    stale+active+clean   [6,8,12]           6   [6,8,12]               6
41.3     stale+active+clean   [8,9,11]           8   [8,9,11]               8
53.1f    stale+active+clean  [9,11,12]           9  [9,11,12]               9
41.0     stale+active+clean   [8,11,9]           8   [8,11,9]               8
53.1c    stale+active+clean   [8,6,11]           8   [8,6,11]               8
41.1     stale+active+clean  [12,11,9]          12  [12,11,9]              12
53.1d    stale+active+clean   [8,11,9]           8   [8,11,9]               8
41.3c    stale+active+clean   [6,9,12]           6   [6,9,12]               6
53.20    stale+active+clean   [9,11,6]           9   [9,11,6]               9
41.3d    stale+active+clean   [9,6,11]           9   [9,6,11]               9
53.21    stale+active+clean  [11,12,8]          11  [11,12,8]              11
41.14    stale+active+clean  [12,11,6]          12  [12,11,6]              12
53.8     stale+active+clean   [8,11,6]           8   [8,11,6]               8
41.2b    stale+active+clean   [8,11,9]           8   [8,11,9]               8
53.3a    stale+active+clean  [12,11,9]          12  [12,11,9]              12
41.12    stale+active+clean   [8,6,11]           8   [8,6,11]               8
53.e     stale+active+clean   [6,11,9]           6   [6,11,9]               6
41.56    stale+active+clean  [12,11,6]          12  [12,11,6]              12
53.38    stale+active+clean  [12,11,6]          12  [12,11,6]              12
53.3e    stale+active+clean    [8,9,6]           8    [8,9,6]               8
41.20    stale+active+clean   [6,11,8]           6   [6,11,8]               6
53.3c    stale+active+clean   [6,11,8]           6   [6,11,8]               6
41.15    stale+active+clean   [6,12,9]           6   [6,12,9]               6
60.0     stale+active+clean   [8,12,6]           8   [8,12,6]               8
41.28    stale+active+clean   [11,8,9]          11   [11,8,9]              11
53.2b    stale+active+clean    [8,6,9]           8    [8,6,9]               8
53.3b    stale+active+clean  [11,9,12]          11  [11,9,12]              11
53.a     stale+active+clean  [11,9,12]          11  [11,9,12]              11
41.2d    stale+active+clean  [8,11,12]           8  [8,11,12]               8
41.31    stale+active+clean  [11,8,12]          11  [11,8,12]              11
41.30    stale+active+clean   [12,9,8]          12   [12,9,8]              12
53.2f    stale+active+clean    [9,6,8]           9    [9,6,8]               9
53.2e    stale+active+clean   [12,9,8]          12   [12,9,8]              12
53.29    stale+active+clean   [9,8,12]           9   [9,8,12]               9
41.39    stale+active+clean  [6,12,11]           6  [6,12,11]               6
41.7e    stale+active+clean   [9,8,12]           9   [9,8,12]               9
53.62    stale+active+clean  [12,11,9]          12  [12,11,9]              12
41.41    stale+active+clean   [9,6,11]           9   [9,6,11]               9
53.5c    stale+active+clean  [12,11,6]          12  [12,11,6]              12
41.42    stale+active+clean  [6,11,12]           6  [6,11,12]               6
53.5e    stale+active+clean   [9,11,8]           9   [9,11,8]               9
41.44    stale+active+clean  [12,9,11]          12  [12,9,11]              12
41.1b    stale+active+clean   [8,12,9]           8   [8,12,9]               8
41.46    stale+active+clean  [11,9,12]          11  [11,9,12]              11
53.5a    stale+active+clean  [12,6,11]          12  [12,6,11]              12
41.1d    stale+active+clean   [12,6,9]          12   [12,6,9]              12
53.56    stale+active+clean    [9,8,6]           9    [9,8,6]               9
41.4d    stale+active+clean  [12,11,8]          12  [12,11,8]              12
41.4c    stale+active+clean   [8,6,12]           8   [8,6,12]               8
41.4f    stale+active+clean   [9,12,6]           9   [9,12,6]               9
53.53    stale+active+clean  [12,6,11]          12  [12,6,11]              12
53.40    stale+active+clean   [11,6,8]          11   [11,6,8]              11
53.46    stale+active+clean   [6,8,11]           6   [6,8,11]               6
41.5a    stale+active+clean  [9,12,11]           9  [9,12,11]               9
53.47    stale+active+clean   [12,6,9]          12   [12,6,9]              12
41.58    stale+active+clean  [12,11,9]          12  [12,11,9]              12
41.1f    stale+active+clean  [11,12,8]          11  [11,12,8]              11
41.59    stale+active+clean    [6,8,9]           6    [6,8,9]               6
53.0     stale+active+clean    [8,6,9]           8    [8,6,9]               8
41.49    stale+active+clean   [8,12,6]           8   [8,12,6]               8
53.55    stale+active+clean    [8,9,6]           8    [8,9,6]               8
53.48    stale+active+clean   [8,11,9]           8   [8,11,9]               8
53.49    stale+active+clean   [8,11,6]           8   [8,11,6]               8
53.74    stale+active+clean   [8,9,12]           8   [8,9,12]               8
41.68    stale+active+clean   [11,6,8]          11   [11,6,8]              11
41.52    stale+active+clean   [12,6,8]          12   [12,6,8]              12
53.4f    stale+active+clean  [9,12,11]           9  [9,12,11]               9
41.53    stale+active+clean   [11,9,8]          11   [11,9,8]              11
53.4d    stale+active+clean    [6,8,9]           6    [6,8,9]               6
53.52    stale+active+clean   [6,11,9]           6   [6,11,9]               6
41.7f    stale+active+clean  [11,9,12]          11  [11,9,12]              11
53.60    stale+active+clean   [6,11,8]           6   [6,11,8]               6
53.66    stale+active+clean  [12,9,11]          12  [12,9,11]              12
41.7a    stale+active+clean  [12,6,11]          12  [12,6,11]              12
41.78    stale+active+clean  [9,11,12]           9  [9,11,12]               9
41.79    stale+active+clean  [12,9,11]          12  [12,9,11]              12
53.69    stale+active+clean   [12,9,8]          12   [12,9,8]              12
53.6e    stale+active+clean   [8,6,11]           8   [8,6,11]               8
41.73    stale+active+clean  [12,9,11]          12  [12,9,11]              12
53.6c    stale+active+clean   [6,12,8]           6   [6,12,8]               6
53.6d    stale+active+clean  [6,11,12]           6  [6,11,12]               6
53.73    stale+active+clean   [8,11,6]           8   [8,11,6]               8
41.6c    stale+active+clean  [12,11,9]          12  [12,11,9]              12
41.6a    stale+active+clean    [9,8,6]           9    [9,8,6]               9
41.67    stale+active+clean   [8,12,9]           8   [8,12,9]               8
53.26    stale+active+clean  [12,11,6]          12  [12,11,6]              12
53.78    stale+active+clean   [8,6,12]           8   [8,6,12]               8
41.3b    stale+active+clean   [8,12,9]           8   [8,12,9]               8
53.79    stale+active+clean    [8,9,6]           8    [8,9,6]               8
41.65    stale+active+clean  [8,12,11]           8  [8,12,11]               8
41.63    stale+active+clean   [8,11,6]           8   [8,11,6]               8
53.7c    stale+active+clean  [12,6,11]          12  [12,6,11]              12
41.60    stale+active+clean    [6,9,8]           6    [6,9,8]               6
41.61    stale+active+clean  [11,8,12]          11  [11,8,12]              11
ok



Bash: 
ceph status
  cluster:
    id:     630e582f-2277-4a8b-a902-8ac08536cd62
    health: HEALTH_WARN
            Reduced data availability: 111 pgs stale
 
  services:
    mon: 3 daemons, quorum pve1,pve2,pve3 (age 2h)
    mgr: pve1(active, since 3d), standbys: pve3, pve2
    mds: 4/4 daemons up, 5 standby
    osd: 11 osds: 11 up (since 3m), 11 in (since 3h); 146 remapped pgs
 
  data:
    volumes: 4/4 healthy
    pools:   13 pools, 1027 pgs
    objects: 4.89M objects, 4.8 TiB
    usage:   12 TiB used, 12 TiB / 24 TiB avail
    pgs:     439719/14684268 objects misplaced (2.994%)
             767 active+clean
             146 active+clean+remapped
             111 stale+active+clean
             3   active+clean+scrubbing+deep
 
Bash: 
ceph osd tree
ID  CLASS  WEIGHT    TYPE NAME       STATUS  REWEIGHT  PRI-AFF
-1         22.75240  root default                             
-3          8.02704      host pve1                           
 7    hdd   3.54648          osd.7       up   1.00000  1.00000
 8    hdd   3.54279          osd.8       up   1.00000  1.00000
 2    ssd   0.93779          osd.2       up   1.00000  1.00000
10    ssd   0.93779          osd.10      up   1.00000  1.00000
-5          7.39078      host pve2                           
 6    hdd   2.75760          osd.6       up   1.00000  1.00000
 9    hdd   2.75760          osd.9       up   1.00000  1.00000
 4    ssd   0.93779          osd.4       up   1.00000  1.00000
 5    ssd   0.93779          osd.5       up   1.00000  1.00000
-6          7.33458      host pve3                           
11    hdd   2.75760          osd.11      up   1.00000  1.00000
12    hdd   2.75760          osd.12      up   1.00000  1.00000
 0    ssd   1.81940          osd.0       up   1.00000  1.00000



Code: 
ceph osd crush tree
ID  CLASS  WEIGHT    TYPE NAME     
-1         22.75240  root default 
-3          8.02704      host pve1
 7    hdd   3.54648          osd.7
 8    hdd   3.54279          osd.8
 2    ssd   0.93779          osd.2
10    ssd   0.93779          osd.10
-5          7.39078      host pve2
 6    hdd   2.75760          osd.6
 9    hdd   2.75760          osd.9
 4    ssd   0.93779          osd.4
 5    ssd   0.93779          osd.5
-6          7.33458      host pve3
11    hdd   2.75760          osd.11
12    hdd   2.75760          osd.12
 0    ssd   1.81940          osd.0
 
You must log in or register to reply here.
Top Bottom