2020-01-28 21:22:36.747643 mon.pve2 mon.0 192.168.66.2:6789/0 255 : cluster [INF] mon.pve2 calling monitor election
2020-01-28 21:22:38.241872 mon.pve3 mon.1 192.168.66.3:6789/0 102 : cluster [INF] mon.pve3 calling monitor election
2020-01-28 21:22:38.409037 mon.pve4 mon.2 192.168.66.4:6789/0 112 : cluster [INF] mon.pve4 calling monitor election
2020-01-28 21:22:39.296906 mon.pve6 mon.4 192.168.66.6:6789/0 124 : cluster [INF] mon.pve6 calling monitor election
2020-01-28 21:22:44.558286 mon.pve2 mon.0 192.168.66.2:6789/0 256 : cluster [INF] mon.pve2 calling monitor election
2020-01-28 21:22:49.635462 mon.pve2 mon.0 192.168.66.2:6789/0 257 : cluster [INF] mon.pve2 is new leader, mons pve2,pve3,pve4,pve5 in quorum (ranks 0,1,2,3)
2020-01-28 21:22:50.565385 mon.pve2 mon.0 192.168.66.2:6789/0 258 : cluster [INF] overall HEALTH_OK
2020-01-28 21:22:50.565456 mon.pve2 mon.0 192.168.66.2:6789/0 259 : cluster [INF] mon.pve2 calling monitor election
2020-01-28 21:22:50.906136 mon.pve2 mon.0 192.168.66.2:6789/0 260 : cluster [INF] mon.pve2 is new leader, mons pve2,pve3,pve4,pve5,pve6 in quorum (ranks 0,1,2,3,4)
2020-01-28 21:22:51.957342 mon.pve2 mon.0 192.168.66.2:6789/0 265 : cluster [INF] overall HEALTH_OK
2020-01-28 21:21:25.815902 osd.34 osd.34 192.168.66.3:6800/4537 1589 : cluster [WRN] slow request 30.704321 seconds old, received at 2020-01-28 21:20:55.111528: osd_op(client.55150877.0:27504209 2.2ea 2:57702b1a:::rbd_data.1a7e4d6b8b4567.00000000000016b3:head [write 2609152~139264] snapc 132=[132] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 15
2020-01-28 21:21:38.268633 osd.16 osd.16 192.168.66.5:6808/5389 1418 : cluster [WRN] slow request 30.115083 seconds old, received at 2020-01-28 21:21:08.153484: osd_op(client.55166360.0:5495520 2.29 2:94224c08:::rbd_data.bda8574b0dc51.000000000001147f:head [write 2613248~8192] snapc 13b=[13b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 21
2020-01-28 21:21:54.738941 osd.34 osd.34 192.168.66.3:6800/4537 1590 : cluster [WRN] 2 slow requests, 1 included below; oldest blocked for > 59.627366 secs
2020-01-28 21:21:54.738949 osd.34 osd.34 192.168.66.3:6800/4537 1591 : cluster [WRN] slow request 30.516713 seconds old, received at 2020-01-28 21:21:24.222180: osd_op(client.47833248.0:27073759 2.2ea 2:57711360:::rbd_data.1703066b8b4567.0000000000017800:head [write 61440~4096] snapc 133=[133] ondisk+write+known_if_redirected e9176) currently op_applied
2020-01-28 21:21:55.736362 osd.34 osd.34 192.168.66.3:6800/4537 1592 : cluster [WRN] 2 slow requests, 1 included below; oldest blocked for > 60.624800 secs
2020-01-28 21:21:55.736370 osd.34 osd.34 192.168.66.3:6800/4537 1593 : cluster [WRN] slow request 60.624800 seconds old, received at 2020-01-28 21:20:55.111528: osd_op(client.55150877.0:27504209 2.2ea 2:57702b1a:::rbd_data.1a7e4d6b8b4567.00000000000016b3:head [write 2609152~139264] snapc 132=[132] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 15
2020-01-28 21:22:09.007044 osd.16 osd.16 192.168.66.5:6808/5389 1419 : cluster [WRN] 1 slow requests, 1 included below; oldest blocked for > 60.853522 secs
2020-01-28 21:22:09.007051 osd.16 osd.16 192.168.66.5:6808/5389 1420 : cluster [WRN] slow request 60.853522 seconds old, received at 2020-01-28 21:21:08.153484: osd_op(client.55166360.0:5495520 2.29 2:94224c08:::rbd_data.bda8574b0dc51.000000000001147f:head [write 2613248~8192] snapc 13b=[13b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 21
2020-01-28 21:22:24.659675 osd.34 osd.34 192.168.66.3:6800/4537 1594 : cluster [WRN] 2 slow requests, 1 included below; oldest blocked for > 89.548109 secs
2020-01-28 21:22:24.659683 osd.34 osd.34 192.168.66.3:6800/4537 1595 : cluster [WRN] slow request 60.437457 seconds old, received at 2020-01-28 21:21:24.222180: osd_op(client.47833248.0:27073759 2.2ea 2:57711360:::rbd_data.1703066b8b4567.0000000000017800:head [write 61440~4096] snapc 133=[133] ondisk+write+known_if_redirected e9176) currently op_applied
2020-01-28 21:21:35.662993 osd.26 osd.26 192.168.66.6:6820/5451 1351 : cluster [WRN] slow request 30.154665 seconds old, received at 2020-01-28 21:21:05.508263: osd_op(client.55166360.0:5495518 2.315 2:a8d8f14d:::rbd_data.bda8574b0dc51.0000000000003f03:head [write 2568192~4096] snapc 13b=[13b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 30
2020-01-28 21:21:55.505470 osd.26 osd.26 192.168.66.6:6820/5451 1352 : cluster [WRN] 3 slow requests, 2 included below; oldest blocked for > 49.997145 secs
2020-01-28 21:21:55.505477 osd.26 osd.26 192.168.66.6:6820/5451 1353 : cluster [WRN] slow request 30.683436 seconds old, received at 2020-01-28 21:21:24.821971: osd_op(client.55255138.0:33649600 2.2c3 2:c34e2ab6:::rbd_data.1b93874b0dc51.0000000000002c7e:head [write 3264512~4096] snapc 14b=[14b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 2
2020-01-28 21:21:55.505481 osd.26 osd.26 192.168.66.6:6820/5451 1354 : cluster [WRN] slow request 30.683309 seconds old, received at 2020-01-28 21:21:24.822098: osd_op(client.55255138.0:33649601 2.2c3 2:c34e2ab6:::rbd_data.1b93874b0dc51.0000000000002c7e:head [write 3272704~12288] snapc 14b=[14b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 2
2020-01-28 21:22:25.268966 osd.26 osd.26 192.168.66.6:6820/5451 1355 : cluster [WRN] 2 slow requests, 2 included below; oldest blocked for > 60.446940 secs
2020-01-28 21:22:25.268975 osd.26 osd.26 192.168.66.6:6820/5451 1356 : cluster [WRN] slow request 60.446940 seconds old, received at 2020-01-28 21:21:24.821971: osd_op(client.55255138.0:33649600 2.2c3 2:c34e2ab6:::rbd_data.1b93874b0dc51.0000000000002c7e:head [write 3264512~4096] snapc 14b=[14b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 2
2020-01-28 21:22:25.268979 osd.26 osd.26 192.168.66.6:6820/5451 1357 : cluster [WRN] slow request 60.446813 seconds old, received at 2020-01-28 21:21:24.822098: osd_op(client.55255138.0:33649601 2.2c3 2:c34e2ab6:::rbd_data.1b93874b0dc51.0000000000002c7e:head [write 3272704~12288] snapc 14b=[14b] ondisk+write+known_if_redirected e9176) currently sub_op_commit_rec from 2
2020-01-28 21:23:03.815322 mon.pve2 mon.0 192.168.66.2:6789/0 267 : cluster [WRN] Health check failed: 3 slow requests are blocked > 32 sec. Implicated osds 16,26 (REQUEST_SLOW)
2020-01-28 21:23:13.443685 mon.pve2 mon.0 192.168.66.2:6789/0 271 : cluster [WRN] Health check update: 2 slow requests are blocked > 32 sec. Implicated osds 26 (REQUEST_SLOW)
2020-01-28 21:23:31.294486 mon.pve2 mon.0 192.168.66.2:6789/0 277 : cluster [INF] Health check cleared: REQUEST_SLOW (was: 2 slow requests are blocked > 32 sec. Implicated osds 26)
2020-01-28 21:23:31.294541 mon.pve2 mon.0 192.168.66.2:6789/0 278 : cluster [INF] Cluster is now healthy