My 4 node cluster has been rock solid for 6 months, until now.
The crash reports are not that revealing
This is happening on 2 nodes, pve02 and pve03. Any idea how to recover from this?
Bash:
$ root@pve02:~# ceph -s
cluster:
id: 3e788c55-0a22-4edc-af28-94b8e4ff1cac
health: HEALTH_WARN
Degraded data redundancy: 1105460/3316380 objects degraded (33.333%), 193 pgs degraded, 193 pgs undersized
12 daemons have recently crashed
services:
mon: 3 daemons, quorum pve01,pve03,pve02 (age 80m)
mgr: pve01(active, since 45h), standbys: pve03, pve02
mds: 1/1 daemons up, 2 standby
osd: 4 osds: 2 up (since 9h), 2 in (since 22m)
data:
volumes: 1/1 healthy
pools: 4 pools, 193 pgs
objects: 1.11M objects, 1.5 TiB
usage: 3.0 TiB used, 696 GiB / 3.6 TiB avail
pgs: 1105460/3316380 objects degraded (33.333%)
193 active+undersized+degraded
io:
client: 9.7 KiB/s wr, 0 op/s rd, 1 op/s wr
Bash:
$ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 7.27759 root default
-3 1.81940 host pve01
0 nvme 1.81940 osd.0 up 1.00000 1.00000
-5 1.81940 host pve02
1 nvme 1.81940 osd.1 down 0 1.00000
-7 1.81940 host pve03
2 nvme 1.81940 osd.2 down 0 1.00000
-9 1.81940 host pve04
3 nvme 1.81940 osd.3 up 1.00000 1.00000The crash reports are not that revealing
Bash:
ceph crash info 2024-11-07T10:37:23.263067Z_3e208c58-2cd5-409f-ab96-e13ea861a27a
{
"assert_condition": "false",
"assert_file": "./src/os/bluestore/HybridAllocator.cc",
"assert_func": "HybridAllocator::init_rm_free(uint64_t, uint64_t)::<lambda(uint64_t, uint64_t, bool)>",
"assert_line": 178,
"assert_msg": "./src/os/bluestore/HybridAllocator.cc: In function 'HybridAllocator::init_rm_free(uint64_t, uint64_t)::<lambda(uint64_t, uint64_t, bool)>' thread 78e76870d840 time 2024-11-07T10:37:23.250154+0000\n./src/os/bluestore/HybridAllocator.cc: 178: FAILED ceph_assert(false)\n",
"assert_thread_name": "ceph-osd",
"backtrace": [
"/lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x78e76925b050]",
"/lib/x86_64-linux-gnu/libc.so.6(+0x8ae3c) [0x78e7692a9e3c]",
"gsignal()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x185) [0x5b4b3ec42362]",
"/usr/bin/ceph-osd(+0x6334a2) [0x5b4b3ec424a2]",
"/usr/bin/ceph-osd(+0xd70dd7) [0x5b4b3f37fdd7]",
"(AvlAllocator::_try_remove_from_tree(unsigned long, unsigned long, std::function<void (unsigned long, unsigned long, bool)>)+0x230) [0x5b4b3f3724e0]",
"(HybridAllocator::init_rm_free(unsigned long, unsigned long)+0xc9) [0x5b4b3f3800a9]",
"(BlueFS::mount()+0x1e9) [0x5b4b3f353269]",
"(BlueStore::_open_bluefs(bool, bool)+0x2dd) [0x5b4b3f2551fd]",
"(BlueStore::_prepare_db_environment(bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)+0x27c) [0x5b4b3f255d1c]",
"(BlueStore::_open_db(bool, bool, bool)+0x37c) [0x5b4b3f26e71c]",
"(BlueStore::_open_db_and_around(bool, bool)+0x48e) [0x5b4b3f2d6c4e]",
"(BlueStore::_mount()+0x347) [0x5b4b3f2d9017]",
"(OSD::init()+0x4b1) [0x5b4b3ed9e3d1]",
"main()",
"/lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x78e76924624a]",
"__libc_start_main()",
"_start()"
],
"ceph_version": "18.2.4",
"crash_id": "2024-11-07T10:37:23.263067Z_3e208c58-2cd5-409f-ab96-e13ea861a27a",
"entity_name": "osd.1",
"os_id": "12",
"os_name": "Debian GNU/Linux 12 (bookworm)",
"os_version": "12 (bookworm)",
"os_version_id": "12",
"process_name": "ceph-osd",
"stack_sig": "5b64dfaf5da18c15ac63e445825a3bfa2cfaab78c531a1ef2cd84f73ebfad950",
"timestamp": "2024-11-07T10:37:23.263067Z",
"utsname_hostname": "pve02",
"utsname_machine": "x86_64",
"utsname_release": "6.8.12-3-pve",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP PREEMPT_DYNAMIC PMX 6.8.12-3 (2024-10-23T11:41Z)"
}This is happening on 2 nodes, pve02 and pve03. Any idea how to recover from this?
