I've got an odd issue that I can't seem to figure out a solution to.
My cluster had 10 hosts, 7 of which participate in Ceph which each have 4x 1.6TB NVMe WD Ultrastar DC SN200 (HUSMR7616BDP301) and 6x 3.84TB SATA Micron 5200 ECO (MTFDDAK3T8TDC) connected via a SAS HBA (no HW RAID present at all).
After joining an 11th host and installing Ceph, I created new OSDs on 2x 15.36TB Intel D7-P5520 (SSDPF2KX153T1). The idea is to phase more of these large NVMe drives in and phase out the SATA drives while we are growing out cluster.
As soon as those one of those new OSDs came online and rebalance to them started the cluster threw a fit. Hundreds of PGs inactive, thousands of SLOW_OPS. Marking the Intel D7 OSDs out resolves the issues temporarily, but I still need to get these added and re-balanced onto without locking up the cluster in the meantime.
Tuning changes I have tried are reducing osd_recovery_max_active_ssd from 10 to 1 and changing osd_mclock_profile from balanced to high_client_ops. It still exhibits the same behavior when marking one of the Intel D7 OSDs in.
ceph health detail with new drives in:
My cluster had 10 hosts, 7 of which participate in Ceph which each have 4x 1.6TB NVMe WD Ultrastar DC SN200 (HUSMR7616BDP301) and 6x 3.84TB SATA Micron 5200 ECO (MTFDDAK3T8TDC) connected via a SAS HBA (no HW RAID present at all).
After joining an 11th host and installing Ceph, I created new OSDs on 2x 15.36TB Intel D7-P5520 (SSDPF2KX153T1). The idea is to phase more of these large NVMe drives in and phase out the SATA drives while we are growing out cluster.
As soon as those one of those new OSDs came online and rebalance to them started the cluster threw a fit. Hundreds of PGs inactive, thousands of SLOW_OPS. Marking the Intel D7 OSDs out resolves the issues temporarily, but I still need to get these added and re-balanced onto without locking up the cluster in the meantime.
Tuning changes I have tried are reducing osd_recovery_max_active_ssd from 10 to 1 and changing osd_mclock_profile from balanced to high_client_ops. It still exhibits the same behavior when marking one of the Intel D7 OSDs in.
ceph health detail with new drives in:
Code:
HEALTH_WARN 2 OSD(s) experiencing slow operations in BlueStore; Reduced data availability: 178 pgs inactive; Degraded data redundancy: 13/149100090 objects degraded (0.000%), 8 pgs degraded; 6355 slow ops, oldest one blocked for 2722 sec, daemons [osd.0,osd.1,osd.10,osd.12,osd.13,osd.17,osd.18,osd.19,osd.22,osd.23]... have slow ops.
[WRN] BLUESTORE_SLOW_OP_ALERT: 2 OSD(s) experiencing slow operations in BlueStore
osd.19 observed slow operation indications in BlueStore
osd.39 observed slow operation indications in BlueStore
[WRN] PG_AVAILABILITY: Reduced data availability: 178 pgs inactive
pg 9.dc is stuck inactive for 45m, current state activating+remapped, last acting [62,37,35,54,7,53,45]
pg 9.eb is stuck inactive for 44m, current state activating+remapped, last acting [60,5,37,15,47,35,59]
pg 9.ed is stuck inactive for 44m, current state activating+remapped, last acting [17,60,2,58,12,38,30]
pg 9.ee is stuck inactive for 44m, current state activating+remapped, last acting [22,11,31,49,62,27,36]
pg 9.f0 is stuck inactive for 44m, current state activating+remapped, last acting [37,66,31,67,6,42,19]
pg 9.f5 is stuck inactive for 45m, current state activating+remapped, last acting [53,19,63,51,38,31,17]
pg 19.e8 is stuck inactive for 44m, current state activating+remapped, last acting [40,13,29,24,44,50,35]
pg 19.ea is stuck inactive for 44m, current state activating+remapped, last acting [17,58,68,60,45,49,13]
pg 19.ef is stuck inactive for 44m, current state activating+remapped, last acting [68,13,8,45,1,57,23]
pg 19.f5 is stuck inactive for 44m, current state activating+remapped, last acting [68,36,18,60,62,3,52]
pg 19.f8 is stuck inactive for 45m, current state activating+remapped, last acting [19,22,59,1,17,65,54]
pg 19.f9 is stuck inactive for 44m, current state activating+remapped, last acting [57,46,59,2,65,34,19]
pg 19.fc is stuck inactive for 45m, current state activating+remapped, last acting [69,54,5,44,66,42,21]
pg 19.119 is stuck inactive for 14m, current state activating+remapped, last acting [1,41,31,39,18,17,59]
pg 19.12c is stuck inactive for 45m, current state activating+remapped, last acting [58,59,12,64,40,49,67]
pg 19.136 is stuck inactive for 44m, current state activating+remapped, last acting [34,44,23,4,19,42,38]
pg 19.137 is stuck inactive for 45m, current state activating+remapped, last acting [69,66,49,58,68,50,39]
pg 19.13d is stuck inactive for 44m, current state activating+remapped, last acting [63,67,43,19,65,52,34]
pg 19.141 is stuck inactive for 45m, current state activating+remapped, last acting [59,49,58,67,14,47,43]
pg 19.142 is stuck inactive for 44m, current state activating+remapped, last acting [67,55,11,31,38,5,56]
pg 19.149 is stuck inactive for 44m, current state activating+remapped, last acting [39,29,55,63,58,43,32]
pg 19.14e is stuck inactive for 45m, current state activating+remapped, last acting [17,67,33,56,19,18,26]
pg 19.153 is stuck inactive for 44m, current state activating+remapped, last acting [40,14,52,17,35,65,60]
pg 19.157 is stuck inactive for 44m, current state activating+remapped, last acting [63,48,43,51,30,66,39]
pg 19.159 is stuck inactive for 44m, current state activating+remapped, last acting [59,64,39,63,58,68,62]
pg 19.15a is stuck inactive for 44m, current state activating+remapped, last acting [32,24,18,59,62,42,36]
pg 19.15d is stuck inactive for 44m, current state activating+remapped, last acting [56,18,43,24,68,59,69]
pg 19.168 is stuck inactive for 44m, current state activating+remapped, last acting [13,5,63,25,39,47,9]
pg 19.17d is stuck inactive for 45m, current state activating+remapped, last acting [17,52,60,34,18,16,19]
pg 19.17e is stuck inactive for 45m, current state activating+remapped, last acting [69,66,18,34,2,5,39]
pg 19.183 is stuck inactive for 44m, current state activating+remapped, last acting [44,62,43,30,60,45,42]
pg 19.188 is stuck inactive for 45m, current state activating+remapped, last acting [58,68,6,23,12,29,1]
pg 19.192 is stuck inactive for 45m, current state activating+remapped, last acting [38,24,69,44,56,32,50]
pg 19.193 is stuck inactive for 45m, current state activating+remapped, last acting [69,47,64,67,49,37,29]
pg 19.1b1 is stuck inactive for 44m, current state activating+remapped, last acting [32,69,42,65,36,52,39]
pg 19.1b8 is stuck inactive for 45m, current state activating+remapped, last acting [59,57,35,55,58,60,34]
pg 19.1bd is stuck inactive for 44m, current state activating+remapped, last acting [41,40,4,60,58,29,16]
pg 19.1bf is stuck inactive for 45m, current state activating+remapped, last acting [51,53,29,55,2,6,54]
pg 19.1c8 is stuck inactive for 45m, current state activating+remapped, last acting [58,30,16,24,17,66,69]
pg 19.1c9 is stuck inactive for 44m, current state activating+remapped, last acting [50,54,19,3,52,67,18]
pg 19.1cb is stuck inactive for 45m, current state activating+remapped, last acting [62,34,28,22,8,7,16]
pg 19.1ce is stuck inactive for 44m, current state activating+remapped, last acting [48,58,6,33,66,56,39]
pg 19.1cf is stuck inactive for 44m, current state activating+remapped, last acting [6,10,19,24,54,63,26]
pg 19.1d1 is stuck inactive for 44m, current state activating+remapped, last acting [38,60,69,42,64,44,40]
pg 19.1da is stuck inactive for 44m, current state activating+remapped, last acting [38,12,49,61,64,46,37]
pg 19.1ed is stuck inactive for 44m, current state activating+remapped, last acting [54,14,67,58,57,28,63]
pg 19.1f0 is stuck inactive for 45m, current state activating+remapped, last acting [35,28,43,46,40,12,18]
pg 19.1f4 is stuck inactive for 45m, current state activating+remapped, last acting [35,19,65,52,34,6,60]
pg 19.1f6 is stuck inactive for 45m, current state activating+remapped, last acting [59,14,54,64,65,39,0]
pg 19.1fb is stuck inactive for 44m, current state activating+remapped, last acting [69,66,16,60,8,33,57]
pg 19.1fe is stuck inactive for 45m, current state activating+remapped, last acting [59,36,3,65,24,31,48]
[WRN] PG_DEGRADED: Degraded data redundancy: 13/149100090 objects degraded (0.000%), 8 pgs degraded
pg 6.19 is activating+degraded, acting [17,70,10]
pg 12.3 is activating+degraded, acting [58,70,67]
pg 12.14 is activating+degraded, acting [28,70,56]
pg 12.1f is activating+degraded, acting [41,56,70]
pg 14.16 is activating+degraded, acting [56,70,52]
pg 16.17 is activating+degraded, acting [43,70,30]
pg 16.1c is activating+degraded, acting [46,70,66]
pg 18.1 is activating+degraded, acting [55,29,70]
[WRN] SLOW_OPS: 6355 slow ops, oldest one blocked for 2722 sec, daemons [osd.0,osd.1,osd.10,osd.12,osd.13,osd.17,osd.18,osd.19,osd.22,osd.23]... have slow ops.