[SOLVED] Cannot restart or delete mon on crashed node

[hape]

Hello,

after a crash (powerloss) of one of three CEPH-Cluster-Nodes i cannot restart or delete (and recreate) the mon of this host.

If i try to remove it i get the message "got timeout".

Ho can i force the deleting of this mon entry in the cluster, to afterwards recreate it?

 

[Moayad]

Hi,

How you try to delete the mon?

Could you please check the `ceph status` output?

 

[hape]

Hi,

How you try to delete the mon?

Could you please check the `ceph status` output?

I get this message even if i delete the mon on WebIF or on CLI.

On the failing node (virtfarm-pgp-c2-3) i get no output from that command. The command is hanging till i interrupt it with [Ctrl]-C.

On the remaining nodes (virtfarm-pgp-c2-1 & virtfarm-pgp-c2-2) i get the following output:

root@virtfarm-pgp-c2-2:/var/lib/ceph/mon# ceph status
  cluster:
    id:     7c09213c-baf3-45fe-9545-e063d9c46e14
    health: HEALTH_WARN
            1/3 mons down, quorum virtfarm-pgp-c2-1,virtfarm-pgp-c2-2
            Degraded data redundancy: 165411/496233 objects degraded (33.333%), 81 pgs degraded, 81 pgs undersized
 
  services:
    mon: 3 daemons, quorum virtfarm-pgp-c2-1,virtfarm-pgp-c2-2 (age 4h), out of quorum: virtfarm-pgp-c2-3
    mgr: virtfarm-pgp-c2-1(active, since 16h), standbys: virtfarm-pgp-c2-2
    mds: 1/1 daemons up, 2 standby
    osd: 3 osds: 2 up (since 4h), 2 in (since 4h)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 81 pgs
    objects: 165.41k objects, 644 GiB
    usage:   1.7 TiB used, 2.4 TiB / 4.1 TiB avail
    pgs:     165411/496233 objects degraded (33.333%)
             81 active+undersized+degraded
 
  io:
    client:   0 B/s rd, 42 KiB/s wr, 0 op/s rd, 2 op/s wr

 

[hape]

I get this message even if i delete the mon on WebIF or on CLI.

On the failing node (virtfarm-pgp-c2-3) i get no output from that command. The command is hanging till i interrupt it with [Ctrl]-C.

On the remaining nodes (virtfarm-pgp-c2-1 & virtfarm-pgp-c2-2) i get the following output:

root@virtfarm-pgp-c2-2:/var/lib/ceph/mon# ceph status
  cluster:
    id:     7c09213c-baf3-45fe-9545-e063d9c46e14
    health: HEALTH_WARN
            1/3 mons down, quorum virtfarm-pgp-c2-1,virtfarm-pgp-c2-2
            Degraded data redundancy: 165411/496233 objects degraded (33.333%), 81 pgs degraded, 81 pgs undersized
 
  services:
    mon: 3 daemons, quorum virtfarm-pgp-c2-1,virtfarm-pgp-c2-2 (age 4h), out of quorum: virtfarm-pgp-c2-3
    mgr: virtfarm-pgp-c2-1(active, since 16h), standbys: virtfarm-pgp-c2-2
    mds: 1/1 daemons up, 2 standby
    osd: 3 osds: 2 up (since 4h), 2 in (since 4h)
 
  data:
    volumes: 1/1 healthy
    pools:   4 pools, 81 pgs
    objects: 165.41k objects, 644 GiB
    usage:   1.7 TiB used, 2.4 TiB / 4.1 TiB avail
    pgs:     165411/496233 objects degraded (33.333%)
             81 active+undersized+degraded
 
  io:
    client:   0 B/s rd, 42 KiB/s wr, 0 op/s rd, 2 op/s wr

On the failing node i see the following messages in syslog all the time writing:

4-04-25T14:45:33.186903+02:00 virtfarm-pgp-c2-3 ceph-mds[1505]: 2024-04-25T14:45:33.184+0200 7a492ae006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:33.186501+0200)
2024-04-25T14:45:33.868261+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:33.866+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:33.867483+0200)
2024-04-25T14:45:34.187579+02:00 virtfarm-pgp-c2-3 ceph-mds[1505]: 2024-04-25T14:45:34.185+0200 7a492ae006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:34.186741+0200)
2024-04-25T14:45:34.188895+02:00 virtfarm-pgp-c2-3 ceph-mds[1505]: 2024-04-25T14:45:34.186+0200 7a492fe006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:34.188734+0200)
2024-04-25T14:45:34.868625+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:34.866+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:34.867790+0200)
2024-04-25T14:45:35.868861+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:35.867+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:35.868068+0200)
2024-04-25T14:45:36.869128+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:36.867+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:36.868345+0200)
2024-04-25T14:45:37.869212+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:37.866+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:37.868578+0200)
2024-04-25T14:45:38.688090+02:00 virtfarm-pgp-c2-3 ceph-mds[1505]: 2024-04-25T14:45:38.685+0200 7a492ae006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:38.687301+0200)
2024-04-25T14:45:38.869658+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:38.867+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:38.868870+0200)
2024-04-25T14:45:39.688420+02:00 virtfarm-pgp-c2-3 ceph-mds[1505]: 2024-04-25T14:45:39.685+0200 7a492ae006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:39.687588+0200)
2024-04-25T14:45:39.870060+02:00 virtfarm-pgp-c2-3 ceph-osd[1674]: 2024-04-25T14:45:39.868+0200 71423c8006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:39.869390+0200)
2024-04-25T14:45:40.688708+02:00 virtfarm-pgp-c2-3 ceph-mds[1505]: 2024-04-25T14:45:40.686+0200 7a492ae006c0 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-04-25T13:45:40.687883+0200)

 

[VictorSTS]

If the monitor service starts correctly (systemctl status ceph-mon@meshpve01-of.service), check network conectivity for Ceph public network from the failed server to the other two. Ceph commands will not output anything if there's no quorum and the failed server seems that can't reach the other monitors, hence has no quorum.

You can direct commands to the local service socket and check it's status:

ceph daemon /var/run/ceph/ceph-mon.$HOSTNAME.asok mon_status

 

[hape]

If the monitor service starts correctly (systemctl status ceph-mon@meshpve01-of.service), check network conectivity for Ceph public network from the failed server to the other two. Ceph commands will not output anything if there's no quorum and the failed server seems that can't reach the other monitors, hence has no quorum.

You can direct commands to the local service socket and check it's status:

ceph daemon /var/run/ceph/ceph-mon.$HOSTNAME.asok mon_status

The mon-service won't start. This is the problem i think.

Here is the output of "systemctl status ceph-mon@virtfarm-pgp-c2-3.service"

root@virtfarm-pgp-c2-3:/var/run/ceph# systemctl status ceph-mon@virtfarm-pgp-c2-3.service
× ceph-mon@virtfarm-pgp-c2-3.service - Ceph cluster monitor daemon
     Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/ceph-mon@.service.d
             └─ceph-after-pve-cluster.conf
     Active: failed (Result: exit-code) since Thu 2024-04-25 16:44:34 CEST; 13min ago
   Duration: 110ms
   Main PID: 86230 (code=exited, status=1/FAILURE)
        CPU: 108ms

Apr 25 16:44:34 virtfarm-pgp-c2-3 systemd[1]: ceph-mon@virtfarm-pgp-c2-3.service: Scheduled restart job, restart counter is at 7.
Apr 25 16:44:34 virtfarm-pgp-c2-3 systemd[1]: Stopped ceph-mon@virtfarm-pgp-c2-3.service - Ceph cluster monitor daemon.
Apr 25 16:44:34 virtfarm-pgp-c2-3 systemd[1]: ceph-mon@virtfarm-pgp-c2-3.service: Start request repeated too quickly.
Apr 25 16:44:34 virtfarm-pgp-c2-3 systemd[1]: ceph-mon@virtfarm-pgp-c2-3.service: Failed with result 'exit-code'.
Apr 25 16:44:34 virtfarm-pgp-c2-3 systemd[1]: Failed to start ceph-mon@virtfarm-pgp-c2-3.service - Ceph cluster monitor daemon.

 

[daviditty]

Hi,

Try systemctl reset-failed ceph-mon@virtfarm-pgp-c2-3.service

And then systemctl start ceph-mon@virtfarm-pgp-c2-3.service

Probably the mon started too quickly after booting.

 

[hape]

Hi,

my problem is solved.

The reason for my problem was a wrong MTU setup of new in the system popped up network interfaces after the upgrade to "PVE-8.2".

After the reboot of the upgraded system, there where to new NICs in the network-setup. The old ones have to be replaced by the new ones in the OVS-Bridge-Ports. I overlooked that on all physical interfaces there must be the MTU changed to the usual size used in my PUBLIC-CEPH-NETWORK of 9000 Bytes.

After i changed this to the correct MTU on all interfaces involved, the quorum comes also up on the failing nodes CEPH-interface.

Now the quorum on all the nodes is ok, and the balancing of the OSDs is running.

After the Upgrade to PVE 8.2 there where other interface names of the intel i40e NIC in the network-setup:

prior update nic name: enp5s0f1

after update nic name: enp5s0f1np1

Thanks to all !!!