Ceph OSD keeps failing!

D

Deleted member 33567

Guest
Hi,

Recent updates have made ceph started to act very weird:

we keep loosing one OSD with following from syslog:

Bash: 
2020-10-17 04:28:21.922478 mon.n02-sxb-pve01 (mon.0) 912 : cluster [INF] osd.6 [v2:172.17.1.2:6814/308596,v1:172.17.1.2:6817/308596] boot
2020-10-17 04:28:23.919914 mon.n02-sxb-pve01 (mon.0) 918 : cluster [WRN] Health check update: Degraded data redundancy: 15240/781113 objects degraded (1.951%), 61 pgs degraded (PG_DEGRADED)
2020-10-17 04:28:22.935866 osd.8 (osd.8) 2005 : cluster [INF] 3.18 continuing backfill to osd.6 from (1412'1563816,2102'1566816] MIN to 2102'1566816
2020-10-17 04:28:28.917032 mon.n02-sxb-pve01 (mon.0) 920 : cluster [INF] osd.6 failed (root=default,host=n02-sxb-pve01) (connection refused reported by osd.0)
2020-10-17 04:28:28.934114 mon.n02-sxb-pve01 (mon.0) 928 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
2020-10-17 04:28:29.937247 mon.n02-sxb-pve01 (mon.0) 930 : cluster [WRN] Health check update: Degraded data redundancy: 15233/781113 objects degraded (1.950%), 55 pgs degraded (PG_DEGRADED)
2020-10-17 04:28:34.726454 mon.n02-sxb-pve01 (mon.0) 934 : cluster [INF] osd.8 failed (root=default,host=n03-sxb-pve01) (connection refused reported by osd.3)
2020-10-17 04:28:34.942949 mon.n02-sxb-pve01 (mon.0) 954 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN)
2020-10-17 04:28:37.293519 mon.n02-sxb-pve01 (mon.0) 958 : cluster [WRN] Health check update: Degraded data redundancy: 41466/781113 objects degraded (5.309%), 142 pgs degraded (PG_DEGRADED)
2020-10-17 04:28:39.953418 mon.n02-sxb-pve01 (mon.0) 959 : cluster [WRN] Health check failed: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY)
2020-10-17 04:28:42.007881 mon.n02-sxb-pve01 (mon.0) 962 : cluster [WRN] Health check update: 1 osds down (OSD_DOWN)
2020-10-17 04:28:42.015487 mon.n02-sxb-pve01 (mon.0) 963 : cluster [INF] osd.6 [v2:172.17.1.2:6814/308850,v1:172.17.1.2:6817/308850] boot
2020-10-17 04:28:42.293935 mon.n02-sxb-pve01 (mon.0) 966 : cluster [WRN] Health check update: Degraded data redundancy: 82616/781113 objects degraded (10.577%), 214 pgs degraded (PG_DEGRADED)
2020-10-17 04:28:47.302802 mon.n02-sxb-pve01 (mon.0) 971 : cluster [WRN] Health check update: Degraded data redundancy: 60544/781113 objects degraded (7.751%), 177 pgs degraded (PG_DEGRADED)
2020-10-17 04:28:47.925637 mon.n02-sxb-pve01 (mon.0) 974 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive)
2020-10-17 04:28:48.929895 mon.n02-sxb-pve01 (mon.0) 975 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
2020-10-17 04:28:48.931768 mon.n02-sxb-pve01 (mon.0) 976 : cluster [INF] osd.8 [v2:172.17.1.3:6818/1402458,v1:172.17.1.3:6819/1402458] boot
2020-10-17 04:28:51.334835 mon.n02-sxb-pve01 (mon.0) 981 : cluster [INF] osd.6 failed (root=default,host=n02-sxb-pve01) (connection refused reported by osd.7)
2020-10-17 04:28:51.934397 mon.n02-sxb-pve01 (mon.0) 1009 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)


1602901883343.png

1602901952180.png

There seems to be a issue communicating between the ceph OSDs.

While whenever I want to start osd.6 it will start once when it fails again will give me a:

command '/bin/systemctl start ceph-osd@6' failed: exit code 1

1602902052955.png

ceph -s
1602902087720.png

1602902113017.png
ceph version:

1602902141278.png
 
Please note this worked for months properly no issues only in latest upgrades issues started.

Bash: 
[global]
     auth_client_required = cephx
     auth_cluster_required = cephx
     auth_service_required = cephx
     cluster_network = 172.17.1.2/24
     fsid = 4a5d5cdc-fc64-4ac2-8e14-6cae6ade627a
     mon_allow_pool_delete = true
     mon_host = 172.17.1.2 172.17.1.3 172.17.1.1
     osd_pool_default_min_size = 2
     osd_pool_default_size = 3
     public_network = 172.17.1.2/24
         mon_cluster_log_file_level = info

[client]
     keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
     keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.n01-sxb-pve01]
     host = n01-sxb-pve01
     mds standby for name = pve

[mds.n02-sxb-pve01]
     host = n02-sxb-pve01
     mds_standby_for_name = pve

[mds.n03-sxb-pve01]
     host = n03-sxb-pve01
     mds_standby_for_name = pve

[mon.n01-sxb-pve01]
     public_addr = 172.17.1.1


While the Proxmox HA cluster have ips:

1602909624426.png

I used the same local network for CEPH since this config was working for us since Proxmox 4.


Syslog:

Code: 
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  -3443> 2020-10-17 06:38:26.176 7fb59f01ac80 -1 osd.6 2298 log_to_monitors {default=true}
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  -3322> 2020-10-17 06:38:26.180 7fb598339700 -1 osd.6 2298 set_numa_affinity unable to identify public interface 'vmbr1' numa node: (2) No such file or directory
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:     -1> 2020-10-17 06:38:32.460 7fb582b0e700 -1 /build/ceph-JY24tx/ceph-14.2.11/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, uint64_t, size_t, ceph::bufferlist*, char*)' thread 7fb582b0e700 time 2020-10-17 06:38:32.456695
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]: /build/ceph-JY24tx/ceph-14.2.11/src/os/bluestore/BlueFS.cc: 1662: FAILED ceph_assert(r == 0)
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  ceph version 14.2.11 (21626754f4563baadc6ba5d50b9cbc48a5730a94) nautilus (stable)
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x564f3f4623c8]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  2: (()+0x5115a0) [0x564f3f4625a0]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0xf10) [0x564f3fa87e80]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  4: (BlueRocksRandomAccessFile::Prefetch(unsigned long, unsigned long)+0x2a) [0x564f3fab320a]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  5: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::InitDataBlock()+0x29f) [0x564f40080e3f]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  6: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::FindKeyForward()+0x1c8) [0x564f40081078]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  7: (()+0x10b2a8a) [0x564f40003a8a]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  8: (rocksdb::MergingIterator::Next()+0xb1) [0x564f40096691]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  9: (rocksdb::DBIter::Next()+0x266) [0x564f3ff93696]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  10: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x564f3fef69bd]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  11: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0xd4a) [0x564f3f9a389a]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  12: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x29b) [0x564f3f9a4f3b]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  13: (PGBackend::objects_list_partial(hobject_t const&, int, int, std::vector<hobject_t, std::allocator<hobject_t> >*, hobject_t*)+0x309) [0x564f3f765159]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  14: (PrimaryLogPG::scan_range(int, int, PG::BackfillInterval*, ThreadPool::TPHandle&)+0xfa) [0x564f3f6d02ca]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  15: (PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x7bf) [0x564f3f6d148f]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  16: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x7ee) [0x564f3f719a1e]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  17: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x392) [0x564f3f545f02]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  18: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x564f3f7e9e92]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  19: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x7d7) [0x564f3f561ba7]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  20: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b4) [0x564f3fb2e0c4]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  21: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564f3fb30ad0]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  22: (()+0x7fa3) [0x7fb59fa5efa3]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  23: (clone()+0x3f) [0x7fb59f60e4cf]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:      0> 2020-10-17 06:38:32.464 7fb582b0e700 -1 *** Caught signal (Aborted) **
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  in thread 7fb582b0e700 thread_name:tp_osd_tp
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  ceph version 14.2.11 (21626754f4563baadc6ba5d50b9cbc48a5730a94) nautilus (stable)
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  1: (()+0x12730) [0x7fb59fa69730]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  2: (gsignal()+0x10b) [0x7fb59f54c7bb]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  3: (abort()+0x121) [0x7fb59f537535]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x564f3f462419]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  5: (()+0x5115a0) [0x564f3f4625a0]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  6: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned long, unsigned long, ceph::buffer::v14_2_0::list*, char*)+0xf10) [0x564f3fa87e80]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  7: (BlueRocksRandomAccessFile::Prefetch(unsigned long, unsigned long)+0x2a) [0x564f3fab320a]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  8: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::InitDataBlock()+0x29f) [0x564f40080e3f]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  9: (rocksdb::BlockBasedTableIterator<rocksdb::DataBlockIter, rocksdb::Slice>::FindKeyForward()+0x1c8) [0x564f40081078]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  10: (()+0x10b2a8a) [0x564f40003a8a]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  11: (rocksdb::MergingIterator::Next()+0xb1) [0x564f40096691]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  12: (rocksdb::DBIter::Next()+0x266) [0x564f3ff93696]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x564f3fef69bd]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  14: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0xd4a) [0x564f3f9a389a]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  15: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> >*, ghobject_t*)+0x29b) [0x564f3f9a4f3b]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  16: (PGBackend::objects_list_partial(hobject_t const&, int, int, std::vector<hobject_t, std::allocator<hobject_t> >*, hobject_t*)+0x309) [0x564f3f765159]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  17: (PrimaryLogPG::scan_range(int, int, PG::BackfillInterval*, ThreadPool::TPHandle&)+0xfa) [0x564f3f6d02ca]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  18: (PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x7bf) [0x564f3f6d148f]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  19: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x7ee) [0x564f3f719a1e]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  20: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x392) [0x564f3f545f02]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  21: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x564f3f7e9e92]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  22: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x7d7) [0x564f3f561ba7]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  23: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b4) [0x564f3fb2e0c4]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  24: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564f3fb30ad0]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  25: (()+0x7fa3) [0x7fb59fa5efa3]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  26: (clone()+0x3f) [0x7fb59f60e4cf]
Oct 17 06:38:32 n02-sxb-pve01 ceph-osd[368771]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Oct 17 06:38:32 n02-sxb-pve01 systemd[1]: ceph-osd@6.service: Main process exited, code=killed, status=6/ABRT
Oct 17 06:38:32 n02-sxb-pve01 systemd[1]: ceph-osd@6.service: Failed with result 'signal'.
Oct 17 06:38:32 n02-sxb-pve01 systemd[1]: ceph-osd@6.service: Service RestartSec=100ms expired, scheduling restart.
Oct 17 06:38:32 n02-sxb-pve01 systemd[1]: ceph-osd@6.service: Scheduled restart job, restart counter is at 3.
Oct 17 06:38:32 n02-sxb-pve01 systemd[1]: Stopped Ceph object storage daemon osd.6.
 
Last edited by a moderator:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom