Proxmox 5 and ceph luminous: can't create monitor

[Norman Uittenbogaart]

tried again:

TASK ERROR: monitor filesystem '/var/lib/ceph/mon/ceph-node0' already exist.

did you run these steps before hand?
https://pve.proxmox.com/wiki/Ceph_Server#Ceph_on_Proxmox_VE_5.0

 

[LEX]

yes.
if you do it through the web interface will error as above. if do it using pveceph createmon (i do it using putty) is created normally.

 

[LEX]

ceph 12.2.0 have same problem.

creating /etc/pve/priv/ceph.client.admin.keyring
monmaptool: monmap file /tmp/monmap
monmaptool: generated fsid ac7d1986-e38f-4ba6-9f74-f4c22fba5dfe
epoch 0
fsid ac7d1986-e38f-4ba6-9f74-f4c22fba5dfe
last_changed 2017-09-07 11:08:29.723715
created 2017-09-07 11:08:29.723715
0: 192.168.1.70:6789/0 mon.node0
monmaptool: writing epoch 0 to /tmp/monmap (1 monitors)
*** Caught signal (Illegal instruction) **
 in thread 7f20cf37ef80 thread_name:ceph-mon
 ceph version 12.2.0 (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)
 1: (()+0x9306d4) [0x55fcd3abe6d4]
 2: (()+0x110c0) [0x7f20ce7370c0]
 3: (rocksdb::VersionBuilder::SaveTo(rocksdb::VersionStorageInfo*)+0x871) [0x55fcd3c7e941]
 4: (rocksdb::VersionSet::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool)+0x26bc) [0x55fcd3b65dcc]
 5: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x11f) [0x55fcd3b2cf9f]
 6: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xe40) [0x55fcd3b2ea10]
 7: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DB**)+0x698) [0x55fcd3b30278]
 8: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55fcd36035a8]
 9: (RocksDBStore::create_and_open(std::ostream&)+0xd7) [0x55fcd3604fe7]
 10: (MonitorDBStore::create_and_open(std::ostream&)+0xe3) [0x55fcd34bc913]
 11: (main()+0x7de) [0x55fcd342486e]
 12: (__libc_start_main()+0xf1) [0x7f20cbb4b2b1]
 13: (_start()+0x2a) [0x55fcd34b99ea]
2017-09-07 11:08:30.378074 7f20cf37ef80 -1 *** Caught signal (Illegal instruction) **
 in thread 7f20cf37ef80 thread_name:ceph-mon

 ceph version 12.2.0 (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)
 1: (()+0x9306d4) [0x55fcd3abe6d4]
 2: (()+0x110c0) [0x7f20ce7370c0]
 3: (rocksdb::VersionBuilder::SaveTo(rocksdb::VersionStorageInfo*)+0x871) [0x55fcd3c7e941]
 4: (rocksdb::VersionSet::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool)+0x26bc) [0x55fcd3b65dcc]
 5: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x11f) [0x55fcd3b2cf9f]
 6: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xe40) [0x55fcd3b2ea10]
 7: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DB**)+0x698) [0x55fcd3b30278]
 8: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55fcd36035a8]
 9: (RocksDBStore::create_and_open(std::ostream&)+0xd7) [0x55fcd3604fe7]
 10: (MonitorDBStore::create_and_open(std::ostream&)+0xe3) [0x55fcd34bc913]
 11: (main()+0x7de) [0x55fcd342486e]
 12: (__libc_start_main()+0xf1) [0x7f20cbb4b2b1]
 13: (_start()+0x2a) [0x55fcd34b99ea]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

     0> 2017-09-07 11:08:30.378074 7f20cf37ef80 -1 *** Caught signal (Illegal instruction) **
 in thread 7f20cf37ef80 thread_name:ceph-mon

 ceph version 12.2.0 (36f6c5ea099d43087ff0276121fd34e71668ae0e) luminous (rc)
 1: (()+0x9306d4) [0x55fcd3abe6d4]
 2: (()+0x110c0) [0x7f20ce7370c0]
 3: (rocksdb::VersionBuilder::SaveTo(rocksdb::VersionStorageInfo*)+0x871) [0x55fcd3c7e941]
 4: (rocksdb::VersionSet::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool)+0x26bc) [0x55fcd3b65dcc]
 5: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x11f) [0x55fcd3b2cf9f]
 6: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xe40) [0x55fcd3b2ea10]
 7: (rocksdb::DB::Open(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::DB**)+0x698) [0x55fcd3b30278]
 8: (RocksDBStore::do_open(std::ostream&, bool)+0x908) [0x55fcd36035a8]
 9: (RocksDBStore::create_and_open(std::ostream&)+0xd7) [0x55fcd3604fe7]
 10: (MonitorDBStore::create_and_open(std::ostream&)+0xe3) [0x55fcd34bc913]
 11: (main()+0x7de) [0x55fcd342486e]
 12: (__libc_start_main()+0xf1) [0x7f20cbb4b2b1]
 13: (_start()+0x2a) [0x55fcd34b99ea]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

TASK ERROR: command 'ceph-mon --mkfs -i node0 --monmap /tmp/monmap --keyring /etc/pve/priv/ceph.mon.keyring' failed: got signal 4

 

[fabian]

yes, http://tracker.ceph.com/issues/20529 is still open..

 

[aderumier]

yes, http://tracker.ceph.com/issues/20529 is still open..

It's a rocksdb bug with sse4.2 .
cephs devs have make a pull request in rocksdb github
https://github.com/facebook/rocksdb/pull/2807

 

[LEX]

except in 12.1.4 was not fixed?

 

[fabian]

except in 12.1.4 was not fixed?

nobody said it was.. the bug is still open upstream and there is a proposed fix which has not yet been merged (fully) upstream. it will probably be fixed at the latest in the first minor stable release of Luminous (12.2.1), we might backport the changes earlier if they are confirmed to fix the problem without causing any new ones.

 

[LEX]

Ceph 12.2.1 released. Can you update packages in test repository?

 

[fabian]

12.2.1 packages are available in our ceph test repository, and include a rocksdb fix for compatibility with non-SSE4.2-capable CPUs.