Hello,
I am currently in the process of migrating a 5-node cluster running Proxmox 8 to version 9, and I am encountering some difficulties with Ceph.
I am currently migrating the third node and am having trouble creating its monitor; across the entire cluster, Ceph remains stuck in a leader election loop. Here is a perfect example of what is happening:
Does anyone have any ideas?
I am currently in the process of migrating a 5-node cluster running Proxmox 8 to version 9, and I am encountering some difficulties with Ceph.
I am currently migrating the third node and am having trouble creating its monitor; across the entire cluster, Ceph remains stuck in a leader election loop. Here is a perfect example of what is happening:
Code:
2026-09-30T21:28:32.093+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:32.305+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:32.462+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:32.504+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:32.539+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:32.700+0000 725c05a806c0 1 mon.pve24@3(electing) e37 handle_auth_request failed to assign global_id
2026-09-30T21:28:32.844+0000 725c0a2896c0 -1 mon.pve24@3(electing) e37 get_health_metrics reporting 8 slow ops, oldest is log(1 entries from seq 219401 at 2026-09-30T21:27:44.819611+0000)
2026-09-30T21:28:32.897+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:33.175+0000 725c0ba8c6c0 1 mon.pve24@3(electing) e37 handle_auth_request failed to assign global_id
2026-09-30T21:28:33.359+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:33.376+0000 725c05a806c0 1 mon.pve24@3(electing) e37 handle_auth_request failed to assign global_id
2026-09-30T21:28:33.404+0000 725c07a846c0 1 mon.pve24@3(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T21:28:33.777+0000 725c05a806c0 1 mon.pve24@3(electing) e37 handle_auth_request failed to assign global_id
2026-09-30T21:28:34.578+0000 725c05a806c0 1 mon.pve24@3(electing) e37 handle_auth_request failed to assign global_id
2026-09-30T21:28:37.187+0000 725c07a846c0 0 log_channel(audit) log [DBG] : from='mgr.152084113 X.X.X.X:0/60073545' entity='mgr.pve28' cmd=[{"prefix": "mon metadata", "id": "pve12"}]: dispatch
2026-09-30T21:28:37.191+0000 725c07a846c0 0 mon.pve24@3(electing) e37 handle_command mon_command({"prefix": "mon metadata", "id": "pve14"} v 0)
2026-09-30T21:28:37.191+0000 725c07a846c0 0 log_channel(audit) log [DBG] : from='mgr.152084113 X.X.X.X:0/60073545' entity='mgr.pve28' cmd=[{"prefix": "mon metadata", "id": "pve14"}]: dispatch
2026-09-30T21:28:37.192+0000 725c07a846c0 0 mon.pve24@3(electing) e37 handle_command mon_command({"prefix": "mon metadata", "id": "pve24"} v 0)
2026-09-30T21:28:37.192+0000 725c07a846c0 0 log_channel(audit) log [DBG] : from='mgr.152084113 X.X.X.X:0/60073545' entity='mgr.pve28' cmd=[{"prefix": "mon metadata", "id": "pve24"}]: dispatch
2026-09-30T21:28:37.193+0000 725c07a846c0 0 mon.pve24@3(electing) e37 handle_command mon_command({"prefix": "mon metadata", "id": "pve26"} v 0)
2026-09-30T21:28:37.193+0000 725c07a846c0 0 log_channel(audit) log [DBG] : from='mgr.152084113 X.X.X.X:0/60073545' entity='mgr.pve28' cmd=[{"prefix": "mon metadata", "id": "pve26"}]: dispatch
2026-09-30T21:28:37.194+0000 725c07a846c0 0 mon.pve24@3(electing) e37 handle_command mon_command({"prefix": "mon metadata", "id": "pve28"} v 0)
2026-09-30T21:28:37.194+0000 725c07a846c0 0 log_channel(audit) log [DBG] : from='mgr.152084113 X.X.X.X:0/60073545' entity='mgr.pve28' cmd=[{"prefix": "mon metadata", "id": "pve28"}]: dispatch
2026-09-30T21:28:37.367+0000 725c07a846c0 0 mon.pve24@3(electing) e37 handle_command mon_command({"prefix": "mon metadata", "id": "pve26"} v 0)
2026-09-30T21:28:37.367+0000 725c07a846c0 0 log_channel(audit) log [DBG] : from='mgr.152084113 X.X.X.X:0/60073545' entity='mgr.pve28' cmd=[{"prefix": "mon metadata", "id": "pve26"}]: dispatch
2026-09-30T21:28:37.846+0000 725c0a2896c0 -1 mon.pve24@3(electing) e37 get_health_metrics reporting 737 slow ops, oldest is log(1 entries from seq 219401 at 2026-09-30T21:27:44.819611+0000)
2026-09-30T21:27:44.816810+0000 mon.pve12 (mon.0) 589408 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:49.819540+0000 mon.pve12 (mon.0) 589409 : cluster [INF] mon.pve12 is new leader, mons pve12,pve14,pve28,pve24 in quorum (ranks 0,1,2,3)
2026-09-30T21:27:49.820374+0000 mon.pve12 (mon.0) 589410 : cluster [WRN] Health detail: HEALTH_WARN There are daemons running an older version of ceph; noscrub,nodeep-scrub flag(s) set
2026-09-30T21:27:49.820386+0000 mon.pve12 (mon.0) 589411 : cluster [WRN] [WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph
2026-09-30T21:27:49.820392+0000 mon.pve12 (mon.0) 589412 : cluster [WRN] mon.pve12 mon.pve14 osd.0 osd.1 osd.4 osd.5 osd.6 osd.7 osd.8 osd.9 osd.10 osd.11 mgr.pve12 mgr.pve14 are running an older version of ceph: 19.2.5
2026-09-30T21:27:49.820396+0000 mon.pve12 (mon.0) 589413 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set
2026-09-30T21:27:49.820415+0000 mon.pve12 (mon.0) 589414 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:54.822464+0000 mon.pve12 (mon.0) 589415 : cluster [INF] mon.pve12 is new leader, mons pve12,pve14,pve28,pve24 in quorum (ranks 0,1,2,3)
2026-09-30T21:27:54.823203+0000 mon.pve12 (mon.0) 589416 : cluster [WRN] Health detail: HEALTH_WARN There are daemons running an older version of ceph; noscrub,nodeep-scrub flag(s) set
2026-09-30T21:27:54.823218+0000 mon.pve12 (mon.0) 589417 : cluster [WRN] [WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph
2026-09-30T21:27:54.823223+0000 mon.pve12 (mon.0) 589418 : cluster [WRN] mon.pve12 mon.pve14 osd.0 osd.1 osd.4 osd.5 osd.6 osd.7 osd.8 osd.9 osd.10 osd.11 mgr.pve12 mgr.pve14 are running an older version of ceph: 19.2.5
2026-09-30T21:27:54.823227+0000 mon.pve12 (mon.0) 589419 : cluster [WRN] [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set
2026-09-30T21:27:54.823236+0000 mon.pve12 (mon.0) 589420 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.070736+0000 mon.pve12 (mon.0) 589421 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.072815+0000 mon.pve12 (mon.0) 589422 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.074631+0000 mon.pve12 (mon.0) 589423 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.076869+0000 mon.pve12 (mon.0) 589424 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.079719+0000 mon.pve12 (mon.0) 589425 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.081680+0000 mon.pve12 (mon.0) 589426 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.083474+0000 mon.pve12 (mon.0) 589427 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.085268+0000 mon.pve12 (mon.0) 589428 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.087065+0000 mon.pve12 (mon.0) 589429 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.088929+0000 mon.pve12 (mon.0) 589430 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.090626+0000 mon.pve12 (mon.0) 589431 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.092292+0000 mon.pve12 (mon.0) 589432 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.094022+0000 mon.pve12 (mon.0) 589433 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T21:27:55.095746+0000 mon.pve12 (mon.0) 589434 : cluster [INF] mon.pve12 calling monitor election
2026-09-30T23:27:32.361+0200 7ac0d91f36c0 1 mgr finish mon failed to return metadata for mon.pve26: (2) No such file or directory
2026-09-30T23:28:05.777+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.777+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.779+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601531) init, last seen epoch 2601531, mid-election, bumping
2026-09-30T23:28:05.780+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.780+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.780+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601533) init, last seen epoch 2601533, mid-election, bumping
2026-09-30T23:28:05.780+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.781+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.781+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601535) init, last seen epoch 2601535, mid-election, bumping
2026-09-30T23:28:05.781+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.781+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.781+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601537) init, last seen epoch 2601537, mid-election, bumping
2026-09-30T23:28:05.782+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.782+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.782+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601539) init, last seen epoch 2601539, mid-election, bumping
2026-09-30T23:28:05.782+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.783+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.783+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601541) init, last seen epoch 2601541, mid-election, bumping
2026-09-30T23:28:05.783+0200 74d464a0a6c0 1 mon.pve26@4(electing) e37 collect_metadata md3: no unique device id for md3: fallback method has no model nor serial no unique device path for md3: no symlink to md3 in /dev/disk/by-path
2026-09-30T23:28:05.783+0200 74d464a0a6c0 0 log_channel(cluster) log [INF] : mon.pve26 calling monitor election
2026-09-30T23:28:05.783+0200 74d464a0a6c0 1 paxos.4).electionLogic(2601543) init, last seen epoch 2601543, mid-election, bumpingDoes anyone have any ideas?