Ceph: Why I have only one monitor running on 3 node cluster?

[slavik-f]

Today I installed ProxMox for first time. Reading manuals, trying everything.

I have 3 nodes, each with 0.5T SSD drive.

Installed ProxMox 8.2.2 on every node. Joined into one cluster.
Installed Ceph on every node.
Created OSD on every node.
Create Pool.
Everything with default options.

Now I see that I have 3 monitors, but only one is running:


when I click START on any of the stopped monitor - nothing happens, they're not getting started.

I tried to restart it from CLI and it shows as started:

root@t7820:~# systemctl status ceph-mon@t7820
● ceph-mon@t7820.service - Ceph cluster monitor daemon
     Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/ceph-mon@.service.d
             └─ceph-after-pve-cluster.conf
     Active: active (running) since Wed 2024-05-22 00:35:46 EDT; 11min ago
   Main PID: 119238 (ceph-mon)
      Tasks: 25
     Memory: 33.1M
        CPU: 2.619s
     CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@t7820.service
             └─119238 /usr/bin/ceph-mon -f --cluster ceph --id t7820 --setuser ceph --setgroup ceph

May 22 00:35:46 t7820 systemd[1]: Started ceph-mon@t7820.service - Ceph cluster monitor daemon.

but web UI still shows it as STOPPED.

When I try to check status from CLI, it appears, that I have only one monitor daemon:

root@t7820:~# ceph status
  cluster:
    id:     f3afacfe-3ffa-4405-bf7e-faa0858b6b3e
    health: HEALTH_OK
  services:
    mon: 1 daemons, quorum ds (age 2m)
    mgr: ds(active, since 5h)
    mds: 1/1 daemons up
    osd: 3 osds: 3 up (since 5h), 3 in (since 5h)
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 97 pgs
    objects: 1.19k objects, 2.5 GiB
    usage:   6.9 GiB used, 1.3 TiB / 1.3 TiB avail
    pgs:     97 active+clean

What am I missing?
Should I have all 3 monitors running? How can I do that?

 

[Maximiliano]

Hello, It is hard to tell what is the issue from the info so far,

Could you please post your Ceph config and monmap? You can get them via

cat /etc/pve/ceph.conf

and

ceph mon dump

In general though, it is easier to destroy a monitor and create a new one rather than fixing an existing one. Just make sure to never delete *all* active monitors of a cluster.