Ceph Mon issue during 9.2.2 -> 9.2.4 Upgrade

[d2600hz]

So I'm really not sure how to fix this one. For info I have a 9 node cluster in each zone (and the problem is inherint on all of my zones), and I started upgrading the servers and have caught an issue which means I had to abort.

I have nodes 1,2,3,4,11,12,13,14,15 . ceph-mon on 1,3,11,13 and 15. I upgraded 1 and 2, and then 3 - when 3 came up, it created an election storm on ceph-mon. I have deleted, purged, recreated, tried injecting a monmap manually, but I cannot work out how to sort this particular issue.

So 1,2 and 3 are on 19.2.4, the rest on 19.2.3 currently

So when I try and recreate I get the initial connect:

2026-07-15T22:05:11.518+0000 70594c3976c0  0 log_channel(cluster) log [INF] : mon.pve011-sjc is new leader, mons pve011-sjc,pve013-sjc,pve015-sjc,pve001-sjc in quorum (ranks 0,1,2,3)
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : monmap epoch 58
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : fsid 46c43c19-dc66-4b87-a50f-be598b4ebef6
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : last_changed 2026-07-15T22:05:01.510754+0000
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : created 2024-03-26T00:32:19.684507+0000
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : min_mon_release 19 (squid)
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : election_strategy: 1
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : 0: [v2:10.30.250.11:3300/0,v1:10.30.250.11:6789/0] mon.pve011-sjc
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : 1: [v2:10.30.250.13:3300/0,v1:10.30.250.13:6789/0] mon.pve013-sjc
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : 2: [v2:10.30.250.15:3300/0,v1:10.30.250.15:6789/0] mon.pve015-sjc
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : 3: [v2:10.30.250.1:3300/0,v1:10.30.250.1:6789/0] mon.pve001-sjc
2026-07-15T22:05:11.523+0000 705949b926c0  0 log_channel(cluster) log [DBG] : 4: [v2:10.30.250.3:3300/0,v1:10.30.250.3:6789/0] mon.pve003-sjc

Everything looks fine, I see the rocksdb stuff in the log in pve003-sjc ceph-mon log. Then it starts calling an election and just flips over and over and over:

2026-07-15T22:05:42.314+0000 74dc56bfe6c0  1 paxos.4).electionLogic(0) init, first boot, initializing epoch at 1
2026-07-15T22:05:42.315+0000 74dc56bfe6c0  0 log_channel(cluster) log [INF] : mon.pve003-sjc calling monitor elec
tion
2026-07-15T22:05:42.315+0000 74dc56bfe6c0  1 paxos.4).electionLogic(0) init, first boot, initializing epoch at 1
2026-07-15T22:05:42.316+0000 74dc56bfe6c0  0 log_channel(cluster) log [INF] : mon.pve003-sjc calling monitor elec
tion
2026-07-15T22:05:42.317+0000 74dc56bfe6c0  1 paxos.4).electionLogic(0) init, first boot, initializing epoch at 1
2026-07-15T22:05:42.317+0000 74dc56bfe6c0  0 log_channel(cluster) log [INF] : mon.pve003-sjc calling monitor elec
tion

On the leader (pve011-sjc) you see the following:

2026-07-15T22:05:31.617+0000 705949b926c0  1 mon.pve011-sjc@0(electing) e58 handle_timecheck drop unexpected msg
2026-07-15T22:05:31.617+0000 705949b926c0  1 mon.pve011-sjc@0(electing) e58 handle_timecheck drop unexpected msg
2026-07-15T22:05:31.620+0000 705949b926c0  1 mon.pve011-sjc@0(electing) e58 handle_timecheck drop unexpected msg
2026-07-15T22:05:31.841+0000 705947b8e6c0  1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.329+0000 705947b8e6c0  1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.357+0000 705947b8e6c0  1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.384+0000 705947b8e6c0  1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.531+0000 705947b8e6c0  1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id

And very shortly ceph-mon just stops responding globally while this is happening - no ceph -s etc.

For info I've done this on the GUI, the cli, and done everything I can think of. Any possible recommendations would be most welcome.

This is happening on every zone, same issue. Everything says I shouldn't force an election to the new 19.2.4 ceph-mon. Currently pve011 is running 19.2.3.

Any help would be appreciated.

 

[j.theisen]

Hello @d2600hz

thank you for posting on the forum!

Are the timestamps on these log excerpts accurate and corresponding to the same time frame?
If so your nodes seem to be out of sync and Ceph is really sensitive in that manner.

Please also check free storage space on all the nodes.
Maybe one of them is running low.

Best regards
Jonas

 

[d2600hz]

No, all servers are synced to a local router via chrony, and ceph time-sync-statusis always showing a skew of 0, latency of 0.

All servers have heaps of space.

 

[d2600hz]

We've tried adding a ceph-mon on other nodes - same result. Existing mons are fine, and we're operational, but unable to add any new nodes to it.

We've enabled debug and just see the same thing. mon tries to join, election happens, existing leader wins, new mon ignores, tries to force election, rinse and repeat. Eventually taking down the mon infrastucture with the election storm.