ceph mgr segmentation fault

[ermanishchawla]

I am on proxmox 6.2 with ceph 14.2.9
keeping getting error one of the manager recently crashed

after seeing the logs, could see the segmentation fault.
What could be the reason

2020-05-20 13:55:00.707 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v2: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:02.707 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v3: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:04.707 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v4: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:06.707 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v5: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:08.712 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v6: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:10.712 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v7: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:12.712 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v8: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:14.712 7f181a09e700 0 log_channel(cluster) log [DBG] : pgmap v9: 0 pgs: ; 0 B data, 0 B use
d, 0 B / 0 B avail
2020-05-20 13:55:15.824 7f18298c6700 -1 *** Caught signal (Segmentation fault) **
in thread 7f18298c6700 thread_name:msgr-worker-2

ceph version 14.2.9 (bed944f8c45b9c98485e99b70e11bbcec6f6659a) nautilus (stable)
1: (()+0x12730) [0x7f182b787730]
2: (bool ProtocolV2::append_frame<ceph::msgr::v2::MessageFrame>(ceph::msgr::v2::MessageFrame&)+0x39a) [0x7f1
82ca61e8a]
3: (ProtocolV2::write_message(Message*, bool)+0x4d9) [0x7f182ca45349]
4: (ProtocolV2::write_event()+0x3a5) [0x7f182ca5a6a5]
5: (AsyncConnection::handle_write()+0x43) [0x7f182ca1b933]
6: (EventCenter:rocess_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000
l> >*)+0x135f) [0x7f182ca6d11f]
7: (()+0x5b9fab) [0x7f182ca72fab]
8: (()+0xbbb2f) [0x7f182b42eb2f]
9: (()+0x7fa3) [0x7f182b77cfa3]
10: (clone()+0x3f) [0x7f182b10e4cf]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
-783> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command assert hook 0x55e3309d6
4e0
-782> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command abort hook 0x55e3309d64
e0
-781> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perfcounters_dump hook
0x55e3309d64e0
-780> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command 1 hook 0x55e3309d64e0
-779> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perf dump hook 0x55e330
9d64e0
-778> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perfcounters_schema hoo
k 0x55e3309d64e0
-777> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perf histogram dump hoo
k 0x55e3309d64e0
-776> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command 2 hook 0x55e3309d64e0
-775> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perf schema hook 0x55e3
309d64e0
-774> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perf histogram schema h
ook 0x55e3309d64e0
-773> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command perf reset hook 0x55e33
09d64e0
-772> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config show hook 0x55e3
309d64e0
-771> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config help hook 0x55e3
309d64e0
-770> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config set hook 0x55e33
09d64e0
-769> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config unset hook 0x55e
3309d64e0
-768> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config get hook 0x55e33
09d64e0
-767> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config diff hook 0x55e3
309d64e0
-766> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command config diff get hook 0x
55e3309d64e0
-765> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command log flush hook 0x55e330
9d64e0
-764> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command log dump hook 0x55e3309
d64e0
-763> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command log reopen hook 0x55e33
09d64e0
-762> 2020-05-20 13:54:39.323 7f182ab13dc0 5 asok(0x55e330ad8000) register_command dump_mempools hook 0x55
e331648068
-761> 2020-05-20 13:54:39.327 7f182ab13dc0 10 monclient: get_monmap_and_config
-760> 2020-05-20 13:54:39.331 7f182ab13dc0 10 monclient: build_initial_monmap
-759> 2020-05-20 13:54:39.331 7f182ab13dc0 10 monclient: monmap:
epoch 0
fsid b020e833-3252-416a-b904-40bb4c97af5e
last_changed 2020-05-20 13:54:39.332849
created 2020-05-20 13:54:39.332849
min_mon_release 0 (unknown)
0: v1:172.19.2.32:6789/0 mon.noname-a-legacy
1: v2:172.19.2.32:3300/0 mon.noname-a

 

[PigLover]

Possibly this: https://tracker.ceph.com/issues/42026

I was seeing this same fault periodically on a non-Proxmox (K8s/rook) install of Ceph. Its fixed in Octopus (Ceph 15.x).

 

[Alwin]

But it should arrive in the next maintenance release of Nautilus.
https://github.com/ceph/ceph/pull/33820