I accidentally set the FAILURE DOMAIN of CEPH to OSD.
This time, I ended up shutting down the OSD, and one of the PROXMOX nodes was shut down due to a failure.
Since then, it has been unable to recover.
This time, I ended up shutting down the OSD, and one of the PROXMOX nodes was shut down due to a failure.
Since then, it has been unable to recover.
Bash:
ceph pg dump_stuck stale
PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY
53.42 stale+active+clean [6,8,9] 6 [6,8,9] 6
41.5f stale+active+clean [12,11,9] 12 [12,11,9] 12
41.19 stale+active+clean [8,11,6] 8 [8,11,6] 8
41.10 stale+active+clean [12,9,8] 12 [12,9,8] 12
53.c stale+active+clean [6,12,8] 6 [6,12,8] 6
41.11 stale+active+clean [11,8,12] 11 [11,8,12] 11
53.d stale+active+clean [11,9,8] 11 [11,9,8] 11
53.12 stale+active+clean [6,9,11] 6 [6,9,11] 6
41.a stale+active+clean [6,12,8] 6 [6,12,8] 6
41.b stale+active+clean [8,12,11] 8 [8,12,11] 8
53.14 stale+active+clean [6,12,11] 6 [6,12,11] 6
53.15 stale+active+clean [8,11,6] 8 [8,11,6] 8
41.6 stale+active+clean [9,8,11] 9 [9,8,11] 9
41.4 stale+active+clean [8,9,12] 8 [8,9,12] 8
41.5 stale+active+clean [6,9,8] 6 [6,9,8] 6
53.19 stale+active+clean [8,12,11] 8 [8,12,11] 8
41.2 stale+active+clean [9,12,11] 9 [9,12,11] 9
53.1e stale+active+clean [6,8,12] 6 [6,8,12] 6
41.3 stale+active+clean [8,9,11] 8 [8,9,11] 8
53.1f stale+active+clean [9,11,12] 9 [9,11,12] 9
41.0 stale+active+clean [8,11,9] 8 [8,11,9] 8
53.1c stale+active+clean [8,6,11] 8 [8,6,11] 8
41.1 stale+active+clean [12,11,9] 12 [12,11,9] 12
53.1d stale+active+clean [8,11,9] 8 [8,11,9] 8
41.3c stale+active+clean [6,9,12] 6 [6,9,12] 6
53.20 stale+active+clean [9,11,6] 9 [9,11,6] 9
41.3d stale+active+clean [9,6,11] 9 [9,6,11] 9
53.21 stale+active+clean [11,12,8] 11 [11,12,8] 11
41.14 stale+active+clean [12,11,6] 12 [12,11,6] 12
53.8 stale+active+clean [8,11,6] 8 [8,11,6] 8
41.2b stale+active+clean [8,11,9] 8 [8,11,9] 8
53.3a stale+active+clean [12,11,9] 12 [12,11,9] 12
41.12 stale+active+clean [8,6,11] 8 [8,6,11] 8
53.e stale+active+clean [6,11,9] 6 [6,11,9] 6
41.56 stale+active+clean [12,11,6] 12 [12,11,6] 12
53.38 stale+active+clean [12,11,6] 12 [12,11,6] 12
53.3e stale+active+clean [8,9,6] 8 [8,9,6] 8
41.20 stale+active+clean [6,11,8] 6 [6,11,8] 6
53.3c stale+active+clean [6,11,8] 6 [6,11,8] 6
41.15 stale+active+clean [6,12,9] 6 [6,12,9] 6
60.0 stale+active+clean [8,12,6] 8 [8,12,6] 8
41.28 stale+active+clean [11,8,9] 11 [11,8,9] 11
53.2b stale+active+clean [8,6,9] 8 [8,6,9] 8
53.3b stale+active+clean [11,9,12] 11 [11,9,12] 11
53.a stale+active+clean [11,9,12] 11 [11,9,12] 11
41.2d stale+active+clean [8,11,12] 8 [8,11,12] 8
41.31 stale+active+clean [11,8,12] 11 [11,8,12] 11
41.30 stale+active+clean [12,9,8] 12 [12,9,8] 12
53.2f stale+active+clean [9,6,8] 9 [9,6,8] 9
53.2e stale+active+clean [12,9,8] 12 [12,9,8] 12
53.29 stale+active+clean [9,8,12] 9 [9,8,12] 9
41.39 stale+active+clean [6,12,11] 6 [6,12,11] 6
41.7e stale+active+clean [9,8,12] 9 [9,8,12] 9
53.62 stale+active+clean [12,11,9] 12 [12,11,9] 12
41.41 stale+active+clean [9,6,11] 9 [9,6,11] 9
53.5c stale+active+clean [12,11,6] 12 [12,11,6] 12
41.42 stale+active+clean [6,11,12] 6 [6,11,12] 6
53.5e stale+active+clean [9,11,8] 9 [9,11,8] 9
41.44 stale+active+clean [12,9,11] 12 [12,9,11] 12
41.1b stale+active+clean [8,12,9] 8 [8,12,9] 8
41.46 stale+active+clean [11,9,12] 11 [11,9,12] 11
53.5a stale+active+clean [12,6,11] 12 [12,6,11] 12
41.1d stale+active+clean [12,6,9] 12 [12,6,9] 12
53.56 stale+active+clean [9,8,6] 9 [9,8,6] 9
41.4d stale+active+clean [12,11,8] 12 [12,11,8] 12
41.4c stale+active+clean [8,6,12] 8 [8,6,12] 8
41.4f stale+active+clean [9,12,6] 9 [9,12,6] 9
53.53 stale+active+clean [12,6,11] 12 [12,6,11] 12
53.40 stale+active+clean [11,6,8] 11 [11,6,8] 11
53.46 stale+active+clean [6,8,11] 6 [6,8,11] 6
41.5a stale+active+clean [9,12,11] 9 [9,12,11] 9
53.47 stale+active+clean [12,6,9] 12 [12,6,9] 12
41.58 stale+active+clean [12,11,9] 12 [12,11,9] 12
41.1f stale+active+clean [11,12,8] 11 [11,12,8] 11
41.59 stale+active+clean [6,8,9] 6 [6,8,9] 6
53.0 stale+active+clean [8,6,9] 8 [8,6,9] 8
41.49 stale+active+clean [8,12,6] 8 [8,12,6] 8
53.55 stale+active+clean [8,9,6] 8 [8,9,6] 8
53.48 stale+active+clean [8,11,9] 8 [8,11,9] 8
53.49 stale+active+clean [8,11,6] 8 [8,11,6] 8
53.74 stale+active+clean [8,9,12] 8 [8,9,12] 8
41.68 stale+active+clean [11,6,8] 11 [11,6,8] 11
41.52 stale+active+clean [12,6,8] 12 [12,6,8] 12
53.4f stale+active+clean [9,12,11] 9 [9,12,11] 9
41.53 stale+active+clean [11,9,8] 11 [11,9,8] 11
53.4d stale+active+clean [6,8,9] 6 [6,8,9] 6
53.52 stale+active+clean [6,11,9] 6 [6,11,9] 6
41.7f stale+active+clean [11,9,12] 11 [11,9,12] 11
53.60 stale+active+clean [6,11,8] 6 [6,11,8] 6
53.66 stale+active+clean [12,9,11] 12 [12,9,11] 12
41.7a stale+active+clean [12,6,11] 12 [12,6,11] 12
41.78 stale+active+clean [9,11,12] 9 [9,11,12] 9
41.79 stale+active+clean [12,9,11] 12 [12,9,11] 12
53.69 stale+active+clean [12,9,8] 12 [12,9,8] 12
53.6e stale+active+clean [8,6,11] 8 [8,6,11] 8
41.73 stale+active+clean [12,9,11] 12 [12,9,11] 12
53.6c stale+active+clean [6,12,8] 6 [6,12,8] 6
53.6d stale+active+clean [6,11,12] 6 [6,11,12] 6
53.73 stale+active+clean [8,11,6] 8 [8,11,6] 8
41.6c stale+active+clean [12,11,9] 12 [12,11,9] 12
41.6a stale+active+clean [9,8,6] 9 [9,8,6] 9
41.67 stale+active+clean [8,12,9] 8 [8,12,9] 8
53.26 stale+active+clean [12,11,6] 12 [12,11,6] 12
53.78 stale+active+clean [8,6,12] 8 [8,6,12] 8
41.3b stale+active+clean [8,12,9] 8 [8,12,9] 8
53.79 stale+active+clean [8,9,6] 8 [8,9,6] 8
41.65 stale+active+clean [8,12,11] 8 [8,12,11] 8
41.63 stale+active+clean [8,11,6] 8 [8,11,6] 8
53.7c stale+active+clean [12,6,11] 12 [12,6,11] 12
41.60 stale+active+clean [6,9,8] 6 [6,9,8] 6
41.61 stale+active+clean [11,8,12] 11 [11,8,12] 11
ok
Bash:
ceph status
cluster:
id: 630e582f-2277-4a8b-a902-8ac08536cd62
health: HEALTH_WARN
Reduced data availability: 111 pgs stale
services:
mon: 3 daemons, quorum pve1,pve2,pve3 (age 2h)
mgr: pve1(active, since 3d), standbys: pve3, pve2
mds: 4/4 daemons up, 5 standby
osd: 11 osds: 11 up (since 3m), 11 in (since 3h); 146 remapped pgs
data:
volumes: 4/4 healthy
pools: 13 pools, 1027 pgs
objects: 4.89M objects, 4.8 TiB
usage: 12 TiB used, 12 TiB / 24 TiB avail
pgs: 439719/14684268 objects misplaced (2.994%)
767 active+clean
146 active+clean+remapped
111 stale+active+clean
3 active+clean+scrubbing+deep