ceph-mgr@{nodename}.service always failed

E

egan wu

New Member
Sep 1, 2025
1
0
1
after update last week (2025-08-29)
all 3 nodes ceph-mgr@{nodename}.service become failed
Sep 01 15:16:29 r7525-01 systemd[1]: Started ceph-mgr@r7525-01.service - Ceph cluster manager daemon.
Sep 01 15:16:32 r7525-01 ceph-mgr[610753]: 2025-09-01T15:16:32.991+0800 748c1fc606c0 -1 librbd::image::PreRemoveRequest: 0x602e1f3fb4d0 check_image_watchers: image has watchers - not removing
Sep 01 15:16:33 r7525-01 ceph-mgr[610753]: 2025-09-01T15:16:33.011+0800 748c1f45f6c0 -1 librbd::image::PreRemoveRequest: 0x602e1f3fba70 check_image_watchers: image has watchers - not removing
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: *** Caught signal (Segmentation fault) **
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: in thread 748c1fc606c0 thread_name:io_context_pool
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: ceph version 19.2.3 (ad1eecf4042e0ce72f382f60c97b709fd6f16a51) squid (stable)
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x748c46e49df0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 2: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1598b0) [0x748c485598b0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 3: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a1843) [0x748c485a1843]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 4: _PyType_LookupRef()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 5: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a216b) [0x748c485a216b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 6: PyObject_GetAttr()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 7: _PyEval_EvalFrameDefault()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 8: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1109dd) [0x748c485109dd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 9: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x3d3442) [0x748c487d3442]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 10: /lib/python3/dist-packages/rbd.cpython-313-x86_64-linux-gnu.so(+0xacfed) [0x748c34d4ffed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 11: /lib/librbd.so.1(+0x3cc8ea) [0x748c345cc8ea]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 12: /lib/librbd.so.1(+0x3ccfed) [0x748c345ccfed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 13: /lib/librbd.so.1(+0x3afec6) [0x748c345afec6]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 14: /lib/librbd.so.1(+0x3b0560) [0x748c345b0560]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 15: /lib/librbd.so.1(+0x2cac93) [0x748c344cac93]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 16: /lib/librbd.so.1(+0x12e7bd) [0x748c3432e7bd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 17: /lib/librbd.so.1(+0x2b1c9e) [0x748c344b1c9e]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 18: /lib/librbd.so.1(+0x2b4379) [0x748c344b4379]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 19: /lib/librados.so.2(+0xd2716) [0x748c46ce4716]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 20: /lib/librados.so.2(+0xd3705) [0x748c46ce5705]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 21: /lib/librados.so.2(+0xd3f8a) [0x748c46ce5f8a]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 22: /lib/librados.so.2(+0xea598) [0x748c46cfc598]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 23: /lib/librados.so.2(+0xd7a71) [0x748c46ce9a71]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 24: /lib/librados.so.2(+0xedf63) [0x748c46cfff63]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 25: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe1224) [0x748c470e1224]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 26: /lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x748c46e9cb7b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 27: /lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x748c46f1a7b8]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 2025-09-01T15:17:03.026+0800 748c1fc606c0 -1 *** Caught signal (Segmentation fault) **
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: in thread 748c1fc606c0 thread_name:io_context_pool
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: ceph version 19.2.3 (ad1eecf4042e0ce72f382f60c97b709fd6f16a51) squid (stable)
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x748c46e49df0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 2: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1598b0) [0x748c485598b0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 3: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a1843) [0x748c485a1843]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 4: _PyType_LookupRef()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 5: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a216b) [0x748c485a216b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 6: PyObject_GetAttr()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 7: _PyEval_EvalFrameDefault()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 8: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1109dd) [0x748c485109dd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 9: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x3d3442) [0x748c487d3442]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 10: /lib/python3/dist-packages/rbd.cpython-313-x86_64-linux-gnu.so(+0xacfed) [0x748c34d4ffed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 11: /lib/librbd.so.1(+0x3cc8ea) [0x748c345cc8ea]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 12: /lib/librbd.so.1(+0x3ccfed) [0x748c345ccfed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 13: /lib/librbd.so.1(+0x3afec6) [0x748c345afec6]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 14: /lib/librbd.so.1(+0x3b0560) [0x748c345b0560]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 15: /lib/librbd.so.1(+0x2cac93) [0x748c344cac93]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 16: /lib/librbd.so.1(+0x12e7bd) [0x748c3432e7bd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 17: /lib/librbd.so.1(+0x2b1c9e) [0x748c344b1c9e]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 18: /lib/librbd.so.1(+0x2b4379) [0x748c344b4379]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 19: /lib/librados.so.2(+0xd2716) [0x748c46ce4716]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 20: /lib/librados.so.2(+0xd3705) [0x748c46ce5705]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 21: /lib/librados.so.2(+0xd3f8a) [0x748c46ce5f8a]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 22: /lib/librados.so.2(+0xea598) [0x748c46cfc598]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 23: /lib/librados.so.2(+0xd7a71) [0x748c46ce9a71]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 24: /lib/librados.so.2(+0xedf63) [0x748c46cfff63]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 25: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe1224) [0x748c470e1224]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 26: /lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x748c46e9cb7b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 27: /lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x748c46f1a7b8]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: -4414> 2025-09-01T15:16:32.991+0800 748c1fc606c0 -1 librbd::image::PreRemoveRequest: 0x602e1f3fb4d0 check_image_watchers: image has watchers - not removing
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: -4370> 2025-09-01T15:16:33.011+0800 748c1f45f6c0 -1 librbd::image::PreRemoveRequest: 0x602e1f3fba70 check_image_watchers: image has watchers - not removing
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 0> 2025-09-01T15:17:03.026+0800 748c1fc606c0 -1 *** Caught signal (Segmentation fault) **
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: in thread 748c1fc606c0 thread_name:io_context_pool
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: ceph version 19.2.3 (ad1eecf4042e0ce72f382f60c97b709fd6f16a51) squid (stable)
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x748c46e49df0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 2: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1598b0) [0x748c485598b0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 3: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a1843) [0x748c485a1843]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 4: _PyType_LookupRef()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 5: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a216b) [0x748c485a216b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 6: PyObject_GetAttr()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 7: _PyEval_EvalFrameDefault()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 8: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1109dd) [0x748c485109dd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 9: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x3d3442) [0x748c487d3442]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 10: /lib/python3/dist-packages/rbd.cpython-313-x86_64-linux-gnu.so(+0xacfed) [0x748c34d4ffed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 11: /lib/librbd.so.1(+0x3cc8ea) [0x748c345cc8ea]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 12: /lib/librbd.so.1(+0x3ccfed) [0x748c345ccfed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 13: /lib/librbd.so.1(+0x3afec6) [0x748c345afec6]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 14: /lib/librbd.so.1(+0x3b0560) [0x748c345b0560]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 15: /lib/librbd.so.1(+0x2cac93) [0x748c344cac93]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 16: /lib/librbd.so.1(+0x12e7bd) [0x748c3432e7bd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 17: /lib/librbd.so.1(+0x2b1c9e) [0x748c344b1c9e]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 18: /lib/librbd.so.1(+0x2b4379) [0x748c344b4379]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 19: /lib/librados.so.2(+0xd2716) [0x748c46ce4716]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 20: /lib/librados.so.2(+0xd3705) [0x748c46ce5705]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 21: /lib/librados.so.2(+0xd3f8a) [0x748c46ce5f8a]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 22: /lib/librados.so.2(+0xea598) [0x748c46cfc598]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 23: /lib/librados.so.2(+0xd7a71) [0x748c46ce9a71]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 24: /lib/librados.so.2(+0xedf63) [0x748c46cfff63]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 25: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe1224) [0x748c470e1224]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 26: /lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x748c46e9cb7b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 27: /lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x748c46f1a7b8]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: -4414> 2025-09-01T15:16:32.991+0800 748c1fc606c0 -1 librbd::image::PreRemoveRequest: 0x602e1f3fb4d0 check_image_watchers: image has watchers - not removing
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: -4370> 2025-09-01T15:16:33.011+0800 748c1f45f6c0 -1 librbd::image::PreRemoveRequest: 0x602e1f3fba70 check_image_watchers: image has watchers - not removing
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 0> 2025-09-01T15:17:03.026+0800 748c1fc606c0 -1 *** Caught signal (Segmentation fault) **
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: in thread 748c1fc606c0 thread_name:io_context_pool
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: ceph version 19.2.3 (ad1eecf4042e0ce72f382f60c97b709fd6f16a51) squid (stable)
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3fdf0) [0x748c46e49df0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 2: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1598b0) [0x748c485598b0]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 3: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a1843) [0x748c485a1843]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 4: _PyType_LookupRef()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 5: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1a216b) [0x748c485a216b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 6: PyObject_GetAttr()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 7: _PyEval_EvalFrameDefault()
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 8: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x1109dd) [0x748c485109dd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 9: /lib/x86_64-linux-gnu/libpython3.13.so.1.0(+0x3d3442) [0x748c487d3442]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 10: /lib/python3/dist-packages/rbd.cpython-313-x86_64-linux-gnu.so(+0xacfed) [0x748c34d4ffed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 11: /lib/librbd.so.1(+0x3cc8ea) [0x748c345cc8ea]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 12: /lib/librbd.so.1(+0x3ccfed) [0x748c345ccfed]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 13: /lib/librbd.so.1(+0x3afec6) [0x748c345afec6]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 14: /lib/librbd.so.1(+0x3b0560) [0x748c345b0560]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 15: /lib/librbd.so.1(+0x2cac93) [0x748c344cac93]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 16: /lib/librbd.so.1(+0x12e7bd) [0x748c3432e7bd]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 17: /lib/librbd.so.1(+0x2b1c9e) [0x748c344b1c9e]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 18: /lib/librbd.so.1(+0x2b4379) [0x748c344b4379]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 19: /lib/librados.so.2(+0xd2716) [0x748c46ce4716]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 20: /lib/librados.so.2(+0xd3705) [0x748c46ce5705]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 21: /lib/librados.so.2(+0xd3f8a) [0x748c46ce5f8a]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 22: /lib/librados.so.2(+0xea598) [0x748c46cfc598]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 23: /lib/librados.so.2(+0xd7a71) [0x748c46ce9a71]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 24: /lib/librados.so.2(+0xedf63) [0x748c46cfff63]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 25: /lib/x86_64-linux-gnu/libstdc++.so.6(+0xe1224) [0x748c470e1224]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 26: /lib/x86_64-linux-gnu/libc.so.6(+0x92b7b) [0x748c46e9cb7b]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: 27: /lib/x86_64-linux-gnu/libc.so.6(+0x1107b8) [0x748c46f1a7b8]
Sep 01 15:17:03 r7525-01 ceph-mgr[610753]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Sep 01 15:17:03 r7525-01 systemd[1]: ceph-mgr@r7525-01.service: Main process exited, code=killed, status=11/SEGV
Sep 01 15:17:03 r7525-01 systemd[1]: ceph-mgr@r7525-01.service: Failed with result 'signal'.
Sep 01 15:17:03 r7525-01 systemd[1]: ceph-mgr@r7525-01.service: Consumed 4.572s CPU time, 371.2M memory peak.
Sep 01 15:17:13 r7525-01 systemd[1]: ceph-mgr@r7525-01.service: Scheduled restart job, restart counter is at 3.
Sep 01 15:17:13 r7525-01 systemd[1]: ceph-mgr@r7525-01.service: Start request repeated too quickly.
Sep 01 15:17:13 r7525-01 systemd[1]: ceph-mgr@r7525-01.service: Failed with result 'signal'.
Sep 01 15:17:13 r7525-01 systemd[1]: Failed to start ceph-mgr@r7525-01.service - Ceph cluster manager daemon.
Click to expand...

any suggestions?
thanks
 
You must log in or register to reply here.
Top Bottom