PVE 5.1 Ceph mon fail

[crash.ntech]

Hello, I have 4 nodes cluster with 4 ceph mon/mgr.
Fourth mon crash after ~5 min. There is logs.

root@c1n4:~# /usr/bin/ceph-mon -f --cluster ceph --id c1n4 --setuser ceph --setgroup ceph
2017-12-02 13:29:57.112750 7f8bb16eaf80 -1 mon.c1n4@-1(probing).mgr e212 Failed to load mgr commands: (2) No such file or directory
^C2017-12-02 13:30:04.388128 7f8ba4b5d700 -1 Fail to open '/proc/0/cmdline' error = (2) No such file or directory
2017-12-02 13:30:04.388151 7f8ba4b5d700 -1 received  signal: Interrupt from  PID: 0 task name: <unknown> UID: 0
2017-12-02 13:30:04.388154 7f8ba4b5d700 -1 mon.c1n4@3(peon) e4 *** Got Signal Interrupt ***

Full log attached.

 

[crash.ntech]

more logs

 

[crash.ntech]

I have tried to remove and re-create mon. But have same error

root@c1n4:~# pveceph createmon
admin_socket: exception getting command descriptions: [Errno 111] Connection refused
INFO:ceph-create-keys:ceph-mon admin socket not ready yet.
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'probing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:ceph-mon is not in quorum: u'electing'
INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-osd/ceph.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-rgw/ceph.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-mds/ceph.keyring
INFO:ceph-create-keys:Key exists already: /var/lib/ceph/bootstrap-rbd/ceph.keyring
creating manager directory '/var/lib/ceph/mgr/ceph-c1n4'
creating keys for 'mgr.c1n4'
setting owner for directory

 

[Alwin]

Could you also post a 'pveversion -v' and a 'ceph -s'?

 

[crash.ntech]

That is ceph bug: http://tracker.ceph.com/issues/21770
You can fix is by stopping active mgr, and restart failed mon.

This bug is fixed in 12.2.2