So I'm really not sure how to fix this one. For info I have a 9 node cluster in each zone (and the problem is inherint on all of my zones), and I started upgrading the servers and have caught an issue which means I had to abort.
I have nodes 1,2,3,4,11,12,13,14,15 . ceph-mon on 1,3,11,13 and 15. I upgraded 1 and 2, and then 3 - when 3 came up, it created an election storm on ceph-mon. I have deleted, purged, recreated, tried injecting a monmap manually, but I cannot work out how to sort this particular issue.
So 1,2 and 3 are on 19.2.4, the rest on 19.2.3 currently
So when I try and recreate I get the initial connect:
Everything looks fine, I see the rocksdb stuff in the log in pve003-sjc ceph-mon log. Then it starts calling an election and just flips over and over and over:
On the leader (pve011-sjc) you see the following:
And very shortly ceph-mon just stops responding globally while this is happening - no
For info I've done this on the GUI, the cli, and done everything I can think of. Any possible recommendations would be most welcome.
This is happening on every zone, same issue. Everything says I shouldn't force an election to the new 19.2.4 ceph-mon. Currently pve011 is running 19.2.3.
Any help would be appreciated.
I have nodes 1,2,3,4,11,12,13,14,15 . ceph-mon on 1,3,11,13 and 15. I upgraded 1 and 2, and then 3 - when 3 came up, it created an election storm on ceph-mon. I have deleted, purged, recreated, tried injecting a monmap manually, but I cannot work out how to sort this particular issue.
So 1,2 and 3 are on 19.2.4, the rest on 19.2.3 currently
So when I try and recreate I get the initial connect:
Code:
2026-07-15T22:05:11.518+0000 70594c3976c0 0 log_channel(cluster) log [INF] : mon.pve011-sjc is new leader, mons pve011-sjc,pve013-sjc,pve015-sjc,pve001-sjc in quorum (ranks 0,1,2,3)
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : monmap epoch 58
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : fsid 46c43c19-dc66-4b87-a50f-be598b4ebef6
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : last_changed 2026-07-15T22:05:01.510754+0000
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : created 2024-03-26T00:32:19.684507+0000
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : min_mon_release 19 (squid)
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : election_strategy: 1
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : 0: [v2:10.30.250.11:3300/0,v1:10.30.250.11:6789/0] mon.pve011-sjc
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : 1: [v2:10.30.250.13:3300/0,v1:10.30.250.13:6789/0] mon.pve013-sjc
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : 2: [v2:10.30.250.15:3300/0,v1:10.30.250.15:6789/0] mon.pve015-sjc
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : 3: [v2:10.30.250.1:3300/0,v1:10.30.250.1:6789/0] mon.pve001-sjc
2026-07-15T22:05:11.523+0000 705949b926c0 0 log_channel(cluster) log [DBG] : 4: [v2:10.30.250.3:3300/0,v1:10.30.250.3:6789/0] mon.pve003-sjcEverything looks fine, I see the rocksdb stuff in the log in pve003-sjc ceph-mon log. Then it starts calling an election and just flips over and over and over:
Code:
2026-07-15T22:05:42.314+0000 74dc56bfe6c0 1 paxos.4).electionLogic(0) init, first boot, initializing epoch at 1
2026-07-15T22:05:42.315+0000 74dc56bfe6c0 0 log_channel(cluster) log [INF] : mon.pve003-sjc calling monitor elec
tion
2026-07-15T22:05:42.315+0000 74dc56bfe6c0 1 paxos.4).electionLogic(0) init, first boot, initializing epoch at 1
2026-07-15T22:05:42.316+0000 74dc56bfe6c0 0 log_channel(cluster) log [INF] : mon.pve003-sjc calling monitor elec
tion
2026-07-15T22:05:42.317+0000 74dc56bfe6c0 1 paxos.4).electionLogic(0) init, first boot, initializing epoch at 1
2026-07-15T22:05:42.317+0000 74dc56bfe6c0 0 log_channel(cluster) log [INF] : mon.pve003-sjc calling monitor elec
tionOn the leader (pve011-sjc) you see the following:
Code:
2026-07-15T22:05:31.617+0000 705949b926c0 1 mon.pve011-sjc@0(electing) e58 handle_timecheck drop unexpected msg
2026-07-15T22:05:31.617+0000 705949b926c0 1 mon.pve011-sjc@0(electing) e58 handle_timecheck drop unexpected msg
2026-07-15T22:05:31.620+0000 705949b926c0 1 mon.pve011-sjc@0(electing) e58 handle_timecheck drop unexpected msg
2026-07-15T22:05:31.841+0000 705947b8e6c0 1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.329+0000 705947b8e6c0 1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.357+0000 705947b8e6c0 1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.384+0000 705947b8e6c0 1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_id
2026-07-15T22:05:32.531+0000 705947b8e6c0 1 mon.pve011-sjc@0(electing) e58 handle_auth_request failed to assign global_idAnd very shortly ceph-mon just stops responding globally while this is happening - no
ceph -s etc.For info I've done this on the GUI, the cli, and done everything I can think of. Any possible recommendations would be most welcome.
This is happening on every zone, same issue. Everything says I shouldn't force an election to the new 19.2.4 ceph-mon. Currently pve011 is running 19.2.3.
Any help would be appreciated.