Thanks for the detailed network diagram. Pulling together everything you've shared, I think the picture is fairly clear now -- though some targeted measurements would still help confirm the timing.
The core reason VMs freeze
During Ceph peering, affected PGs are not available for client I/O. Any QEMU I/O request to a disk on a peering PG simply blocks until peering completes. Normally peering takes milliseconds to a few seconds and guests don't notice. In your case, several factors stack up to stretch peering into tens of seconds, long enough to trigger the `jbd2` 120s blocked alarm and eventually cause the guest filesystem to go read-only.
Why peering is so slow in your environment
In a normal node reentry -- no network disruption, healthy topology -- the same 75 PGs would peer and return to `
active+clean` in a few seconds without guests noticing. The key question is what makes this case different.
- cSTP reconvergence disrupts traffic between healthy nodes (ring-specific)
In a star topology (nodes connected through a central switch), a node returning only affects its own uplink. Other nodes keep talking to each other uninterrupted, so peering between healthy OSDs proceeds normally while the returning OSDs bootstrap.
In your ring topology, STP blocks one port somewhere in the ring to prevent a loop. When the returning node's two ports come up, STP reconverges the entire ring — the blocked port may move, and during the transition, traffic between *healthy* nodes can be briefly interrupted too. Your four independent bridge instances (br0, br1, br2, vmbr2) each reconverge separately and asynchronously, so the effective disruption window on br1 (Ceph VLAN) depends on when that specific bridge resolves.
If br1 is disrupted for longer than `osd_heartbeat_grace` (default 20s), the Monitor may mark multiple OSDs down — not just the returning node's, but any whose heartbeats are lost during the disruption — and then back up, triggering a full down→up cycle on top of the reentry peering. If shorter than 20s, peering messages between healthy OSDs are still dropped and must time out and retry, stretching what would normally be a sub-second exchange into tens of seconds per PG.
- Retries across 75 PGs amplify the load on booting OSDs
A single node hosts two OSDs that are primary or acting-set members for 75 PGs. In the normal case, all 75 start peering simultaneously and each completes in milliseconds — the parallelism is fine. What changes here is that STP-induced message loss forces retries. Each of the 75 PGs retries its GetInfo and GetLog requests, and the target of most of those retries is the returning node's OSDs — which are still running BlueStore mount and journal recovery. The combination of bootstrap I/O and a retry storm across all 75 PGs simultaneously is what pushes peering time from seconds into the range that triggers guest SCSI timeouts.
- HA lock loss (separate, concurrent symptom)
The `HA agent lost lock` events you saw are a separate consequence of br0 (Corosync VLAN) reconverging — not a feedback loop through Ceph. Proxmox HA locks through `pmxcfs`/Corosync, not RADOS. The two disruptions happen concurrently because the same node return triggers STP reconvergence on all bridges at once, but they're independent failure modes.
Recommended fixes, in priority order
- Is 4-second convergence sufficient? Tune heartbeat grace to find out
With `bridge-fd 2`, the typical forwarding delay is ~4s after a topology change is detected. However, the worst case in classic STP includes the max-age detection timeout (default 20s) + 2×fd = ~24s. With the default `osd_heartbeat_grace=20s`, your worst-case 24s disruption still causes OSDs to be marked down.
Setting heartbeat grace above your worst-case convergence time prevents the OSD down→up cycle entirely:
Bash:
ceph config set global osd_heartbeat_grace 30 # default 20 — covers your worst-case ~24s
This is not a runtime-dynamic option — both OSDs and monitors need to restart to pick it up. Set it in `global` (not just `osd`) so that monitors read it too, since both daemons use this value. A rolling OSD restart is sufficient; monitors can be restarted one at a time without quorum loss.
With this in place, a 4s typical disruption will cause TCP connection resets and peering message loss, but OSDs won't be declared down. Whether the resulting reconnect overhead and retry storm still pushes peering past the SCSI timeout threshold is the remaining question — that's exactly what the diagnostics below will tell you. If the freeze disappears or shortens significantly, 4s convergence is sufficient. If VMs still freeze, RSTP is necessary.
- If 4-second convergence is not sufficient: move to RSTP
Since `mstpd` is no longer available in Trixie, the practical option is Open vSwitch, which supports RSTP natively and is available as `openvswitch-switch`. RSTP converges in ~1s rather than 4s and eliminates the topology-change detection delay entirely. The trade-off is that OVS adds configuration complexity compared to Linux bridges.
- Reduce peering log comparison work (minor, but easy)
The GetLog phase of peering compares PG logs between OSDs. Reducing the cap reduces the amount of data exchanged per PG:
Bash:
ceph config set osd osd_max_pg_log_entries 1000 # default 10000
This controls how many log entries OSDs retain per PG going forward — it won't retroactively trim already-accumulated logs, so it takes effect gradually as logs are trimmed over time. It won't help with the very next reboot but reduces the steady-state GetLog cost.
- Separate public and cluster networks (best practice, not a fix for this issue)
Your public and cluster networks are on the same subnet. With 10G NICs, peering messages alone won't saturate the link — this is unlikely to be contributing to the freeze. That said, separating them is worth doing eventually so that post-peering recovery and backfill traffic doesn't compete with guest I/O over the long term. It's not urgent here.
Confirming whether fix #2 is needed
Apply fix #1 (`
osd_heartbeat_grace=30`) unconditionally — it's low-risk and prevents the worst case regardless. Then run a controlled reboot to see if the freeze is gone, shortened, or unchanged. These captures help interpret the result:
Quick post-mortem (no preparation needed):
Bash:
# After the freeze resolves — did bridge ports leave forwarding state?
dmesg -T | grep -Ei 'topology|stp|blocking|forwarding|listening|learning|br[0-9]'
# Were OSDs actually marked down/up?
grep -E 'osd\.[0-9]+ (down|up)' /var/log/ceph/ceph.log | tail -40
If OSDs were marked down and then back up despite the new 30s grace period, the STP disruption is exceeding your worst-case estimate — fix #2 (RSTP) becomes necessary. If OSDs stayed up and the freeze is gone or much shorter, fix #1 was sufficient and 4s convergence is tolerable.
Real-time capture (run before triggering the reboot):
Bash:
# Watch OSD up/down events live
ceph -w >> /tmp/ceph-events.log &
# Measure actual packet loss on the Ceph network
tcpdump -i br1 -n 'portrange 6800-7300' -w /tmp/br1-ceph.pcap &
# STP events
dmesg -w | grep -Ei 'topology|stp|blocking|forwarding|listening|learning|br[0-9]' \
>> /tmp/stp-events.log &
# Bridge port state changes
bridge monitor >> /tmp/bridge-monitor.log &
After the freeze, stop with `
kill %1 %2 %3 %4` and and review the logs:
pcap — actual disruption duration on br1:
Bash:
tcpdump -r /tmp/br1-ceph.pcap -n -tt | awk \
'NR==1{prev=$1; next} {gap=$1-prev; if(gap>1) print "GAP: "gap"s at "$1; prev=$1}'
Each line reports a gap in Ceph traffic between healthy nodes. The duration tells you the actual br1 disruption window.
`
/tmp/ceph-events.log` — were OSDs declared down?
Bash:
grep -E 'osd\.[0-9]+ (down|up)' /tmp/ceph-events.log
Look for `
osd.N down` followed by `
osd.N up` events. Note their timestamps relative to when you triggered the reboot. If you see down/up pairs, the disruption exceeded 30s (or the heartbeat grace wasn't picked up yet) and the full down→up cycle occurred.
`
/tmp/bridge-monitor.log` — port-level detail:
Bash:
grep -A2 'br1' /tmp/bridge-monitor.log
`
bridge monitor` outputs netlink events per port. Look for state change events on br1's member interfaces (the physical NICs bound to br1) around the time of the reboot. This gives finer-grained timing than dmesg alone.
The gap duration in the pcap is the key number:
- if it exceeds 30s, OSDs were declared down despite fix #1.
- If it's under 30s but VMs still freeze, TCP reconnect overhead during the disruption is pushing peering past the SCSI timeout — fix #2 (RSTP via OVS) is needed.
