[SOLVED] Ceph HEALTH_WARN: no active mgr

[cmonty14]

Hi,

I get a Ceph Health Warning:
root@ld4257:~# ceph -s
cluster:
id: fda2f219-7355-4c46-b300-8a65b3834761
health: HEALTH_WARN
no active mgr
clock skew detected on mon.ld4464, mon.ld4465

services:
mon: 3 daemons, quorum ld4257,ld4464,ld4465
mgr: no daemons active
osd: 0 osds: 0 up, 0 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 0 kB used, 0 kB / 0 kB avail
pgs:

However I cannot start any ceph-mgr@.service on my 3-node-cluster:
root@ld4257:~# systemctl status ceph-mgr@ld4257.service
● ceph-mgr@ld4257.service - Ceph cluster manager daemon
Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/ceph-mgr@.service.d
└─ceph-after-pve-cluster.conf
Active: activating (auto-restart) (Result: exit-code) since Tue 2018-07-24 09:44:04 CEST; 8s ago
Process: 5474 ExecStart=/usr/bin/ceph-mgr -f --cluster ${CLUSTER} --id ld4257 --setuser ceph --setgrou
Main PID: 5474 (code=exited, status=1/FAILURE)

Jul 24 09:44:04 ld4257 systemd[1]: ceph-mgr@ld4257.service: Failed with result 'exit-code'.


In the relevant log I can see only this:
root@ld4257:~# cat /var/log/ceph/ceph-mgr.ld4257.log
2018-07-24 09:31:19.718407 7fb3c8202700 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-24 09:31:19.718420 7fb3c8202700 0 ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable), process (unknown), pid 2985
2018-07-24 09:31:19.718850 7fb3c8202700 -1 unable to parse network: 10.96.131.1.0/24
2018-07-24 09:31:29.965682 7fa08728d700 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-24 09:31:29.965701 7fa08728d700 0 ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable), process (unknown), pid 3275
2018-07-24 09:31:29.966364 7fa08728d700 -1 unable to parse network: 10.96.131.1.0/24
2018-07-24 09:31:40.214498 7f2f89c58700 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-24 09:31:40.214514 7f2f89c58700 0 ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable), process (unknown), pid 3294
2018-07-24 09:31:40.215064 7f2f89c58700 -1 unable to parse network: 10.96.131.1.0/24
2018-07-24 09:43:54.428352 7f844279f700 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-24 09:43:54.428371 7f844279f700 0 ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable), process (unknown), pid 5442
2018-07-24 09:43:54.428886 7f844279f700 -1 unable to parse network: 10.96.131.1.0/24
2018-07-24 09:44:04.464953 7f2302fd1700 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-24 09:44:04.464967 7f2302fd1700 0 ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable), process (unknown), pid 5474
2018-07-24 09:44:04.465567 7f2302fd1700 -1 unable to parse network: 10.96.131.1.0/24
2018-07-24 09:44:14.715962 7fa37e3bc700 0 set uid:gid to 64045:64045 (ceph:ceph)
2018-07-24 09:44:14.715980 7fa37e3bc700 0 ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable), process (unknown), pid 5489
2018-07-24 09:44:14.716583 7fa37e3bc700 -1 unable to parse network: 10.96.131.1.0/24


I have no clue what's wrong here, but the network 10.96.131.1.0/24 is the public network and I have configured a cluster network.
And there's no issue with the network!

root@ld4257:~# pveversion
pve-manager/5.2-5/eb24855a (running kernel: 4.15.18-1-pve)
root@ld4257:~# ceph --version
ceph version 12.2.5 (dfcb7b53b2e4fcd2a5af0240d4975adc711ab96e) luminous (stable)

 

[Alwin]

unable to parse network: 10.96.131.1.0/24

Five octets? I guess, you wanted to replace the .1 with the .0.

 

[cmonty14]

Wrong configuration /etc/ceph/ceph.conf.

 

[Rongxin Yin]

Wrong configuration /etc/ceph/ceph.conf.

I got the same issue. how do you configure /etc/ceph/ceph.conf.
thanks!

 

[tuxick]

In my case i got it fixed by running on one node:

pveceph mgr destroy pvetest1
pveceph mgr create

and waited a bit. After this the "got timeout(500)" and other timeout issues went away.