PG stuck peering (peering_blocked_by_history_les_bound) after OSD host reboot

[FProxo]

Hi all,

during a rolling PVE 8 -> 9 upgrade on our 3-node hyper-converged cluster, one PG got stuck in peering for ~9 minutes after a node reboot and blocked VM I/O until we ran a manual ceph pg repeer. We dug through the logs and found what looks like a lost message in the OSD messenger. Details below – maybe someone has seen this before.

Environment

• Ceph 19.2.6 (squid), Proxmox VE 9 (kernel 7.0.14-24-pve), during a rolling PVE 8 -> 9 upgrade. At the time of the incident the cluster ran two builds of 19.2.6: Debian bookworm build on the primary's host, trixie build on the rebooted host.
• 3 hosts, 8 OSDs each (24 SSD OSDs), 1 replicated pool, size=3, min_size=2, 512 PGs
• public_network == cluster_network (10.0.50.0/24, 10 GbE via switch, no STP on host bridges)
• noout set, host rebooted for upgrade

What happened
After host node2 (osd.8–15) rebooted, all its OSDs booted at 13:39:53 (osdmap e11417). PG 5.1a0 (up/acting [16,15,6], primary osd.16) stayed in "peering" for ~9 minutes, blocking client I/O (45 slow ops, "waiting for peered" on osd.16).

ceph pg 5.1a0 query showed blocked_by [15] and peering_blocked_by_detail: peering_blocked_by_history_les_bound. All OSDs were up/in and there was no osdmap change after e11422, so peering never restarted by itself. ceph pg repeer 5.1a0 (e11423/e11424) resolved it within ~1 s (active+clean at e11425).

Evidence
osd.16 (primary, on node3) entered the new interval, osd.15 (replica, on node2, just booted) entered the same interval 1 ms later:

2026-10-16T13:39:53.355+0200 osd.16 pg_epoch: 11417 pg[5.1a0(...) sis=11417 [16,15,6] r=0
2026-10-16T13:39:53.356+0200 osd.15 pg_epoch: 11417 pg[5.1a0(...) sis=11417 [16,15,6] r=1

64 ms later osd.15 logged this on its cluster connection from osd.16 (osd.16 cluster addr v2:10.0.50.3:6810/3086584503, confirmed via ceph osd dump 11417):

2026-10-16T13:39:53.419+0200 osd.15 --2- [v2:10.0.50.2:6846/1335120303,...] >> [v2:10.0.50.3:6810/3086584503,...] conn(... crc :-1 s=READ_MESSAGE_COMPLETE pgs=138 gs=27 cs=0 l=0 ... reconnecting=0 rev1=1 ...).handle_message missed message?  skipped from seq 0 to 11

• This is the only messenger warning in all 24 OSD logs during the incident window.
• No further 5.1a0 state changes on any of the three OSDs until the repeer at 13:48:46.
• osd.15 had written to 5.1a0 in the previous interval [11408,11411) (local les=11408/11409, v 11410'22916119), so the primary could not go active without its info.

Ruled out

• Network: NIC link up at 13:39:35 (18 s before OSD boot), no link/bridge events on any host, no heartbeat_check failures, no connection faults in OSD logs
• Clock: 0.077 s mon clock skew right after boot, cleared at 13:40:11, no auth errors
• Disks: no slow BlueStore / RocksDB warnings

Question
Is this a known msgr2 issue (messages 1–10 sent on a connection/session that was replaced during connection setup with a freshly booted peer, never resent)? Is there a fix in a later squid/tentacle release? Debug logs (debug_ms) were not enabled at the time, we can enable them for the next planned reboot if that helps.

Thanks in advance for any hints. Happy to provide more logs or run additional diagnostics.