Corosync logging flooded with "retransmit list" entries — when to worry, when to ignore

[bridman19]

Hi all,

3-node PVE 9.x cluster, dedicated corosync network on a separate 1GbE VLAN (yes
yes, 1GbE for corosync isn't ideal, but it's physically separate so latency is
predictable).

Corosync is healthy, quorum is fine, no VM impact reported. But the logs are
constantly showing things like:

corosync[1234]: [TOTEM] Retransmit List: 4a 4b 4c
corosync[1234]: [TOTEM] Retransmit List: 5d 5e

A few per minute, sometimes a burst of 20-30 in a few seconds, then quiet.

When is this normal background noise vs. actually pre-failure? At what point
should I be worried? I don't want to be the guy who ignored it until corosync
actually flapped, but I also don't want to chase a non-issue.

 

[UdoB]

1GbE for corosync isn't ideal

It is sufficient for one or two (or more?) handful of nodes.

But this:

dedicated corosync network on a separate 1GbE VLAN

just does not work! You need to have a separate wire. In your case any other VLAN can saturate this physical wire and make corosync die. There may be priority "tricks" inside of a switch to try to fight this.

Disclaimer: I am not saying this is your problem...

 

[g.fiore]

+1 to UdoB's point — this trips up a lot of people and it's worth
underlining. "Dedicated VLAN" and "dedicated wire" solve completely
different problems.

A VLAN tag protects you from L2 broadcast domain contamination and gives
the switch something to prioritize on, but on the host side the traffic
still hits the same NIC ring buffer, the same interrupt, and the same
physical link. If a backup job on another VLAN pushes the uplink to
line rate, corosync UDP starts getting delayed or dropped and you see
exactly the retransmit pattern @bridman19 is describing.

Priority queuing on the switch helps but only up to a point — it doesn't
help if the congestion is on the host NIC egress queue rather than in the
switch fabric. And most people who "set up QoS" for corosync forget to
set the DSCP/CoS marking on the corosync packets themselves, so the
switch never actually prioritizes them.

@bridman19: two quick checks that don't require rearchitecting anything,
just to characterize what you're seeing:

# per-interface counters — look for RX drops / errors trending up
ethtool -S <corosync-nic> | grep -E "drop|error|missed|discard"

# check corosync link stats — knet exposes latency and packet loss
corosync-cfgtool -n

The second one is really useful: it shows current latency per link and
whether knet is seeing packet loss. If latency is stable in the low ms
and pong_count keeps climbing without gaps, the retransmits are cosmetic.
If latency spikes or you see connection drops, you have a real problem
that matches UdoB's saturation hypothesis.