corosyn error on syslog

[agapitox]

Hi,

We have a cluster with 10 nodes, all of them with the version 6.2-12 and in two nodes we have observed that these messages appear in the syslog:

Dec 28 18:51:01 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf43
Dec 28 18:51:01 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf44
Dec 28 18:51:03 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf54
Dec 28 18:51:03 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf55
Dec 28 18:51:03 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf56
Dec 28 18:51:03 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf57
Dec 28 18:51:03 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf58
Dec 28 18:51:03 pve10 corosync[19393]:   [TOTEM ] Retransmit List: 5fcf59

I've been checking, and I see everything fine in terms of dns, time synchronization, and I see that both hosts have the same packages

root@pve10:~# pveversion -verbose
proxmox-ve: 6.2-2 (running kernel: 5.4.65-1-pve)
pve-manager: 6.2-12 (running version: 6.2-12/b287dd27)
pve-kernel-5.4: 6.2-7
pve-kernel-helper: 6.2-7
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.34-1-pve: 5.4.34-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-2
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-9
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 0.9.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-1
pve-cluster: 6.2-1
pve-container: 3.2-2
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-1
pve-qemu-kvm: 5.1.0-3
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-15
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.4-pve2

as a curious note, both nodes are connected to the same switch, but in the switch we do not observe any anomalous behaviour

what can cause this?

also in the same cluster, we have another small fault, and that is that if we access the web portal from two specific nodes, one node appears in grey and with the question mark. But both the node and the vm's it contains are ok from the rest of the nodes.

Can there be any relation?

Thank you very much, I expect answers

 

[spirit]

are theses 2 nodes same hardware than other ? (same nic ? same cpu or slower cpu ?)

What is your switch model ?

what is the network latency between nodes ? (a simple "ping -f" between nodes)


do you see this message every second always ? or only sometime ?

the grey nodes in gui is clearly related, if you have always restransmit, the pvestatd daemon will not be able to send/receive stats to/from others nodes)

 

[agapitox]

are theses 2 nodes same hardware than other ? (same nic ? same cpu or slower cpu ?)

What is your switch model ?

what is the network latency between nodes ? (a simple "ping -f" between nodes)


do you see this message every second always ? or only sometime ?

the grey nodes in gui is clearly related, if you have always restransmit, the pvestatd daemon will not be able to send/receive stats to/from others nodes)o

One node: PowerEdge R440 4x16GB 1x Silver 4208 CPU @ 2.10GHz (8 core) 2x 240GB ssd zfs mirror and 4x 480GB ssd raidz1 dual NetXtreme BCM5720 Gigabit Ethernet

The other node: PowerEdge R440 4x64GB 1x Gold 5218 CPU @ 2.30GHz (16core) 8x 800GB ssd radiz2 dual NetXtreme BCM5720 Gigabit Ethernet

The switch is a Cisco WS-C3650-48TQ

Latency between nodes:

from pve8 to pve10:

root@pve8:~# ping -f 192.168.7.15
PING 192.168.7.15 (192.168.7.15) 56(84) bytes of data.
.^C
--- 192.168.7.15 ping statistics ---
203536 packets transmitted, 203535 received, 0.000491314% packet loss, time 199ms
rtt min/avg/max/mdev = 0.067/0.084/3.339/0.018 ms, ipg/ewma 0.094/0.081 ms
root@pve8:~#

from pve10 to pve8:

root@pve10:~# ping -f 192.168.7.11
PING 192.168.7.11 (192.168.7.11) 56(84) bytes of data.
.^C
--- 192.168.7.11 ping statistics ---
161454 packets transmitted, 161453 received, 0.000619371% packet loss, time 410ms
rtt min/avg/max/mdev = 0.060/0.086/3.356/0.014 ms, ipg/ewma 0.095/0.084 ms
root@pve10:~#

from pve9 to pve17 (the pve17 node is the one seen in grey from the pve9 node):

root@pve9:~# ping -f 192.168.7.19
PING 192.168.7.19 (192.168.7.19) 56(84) bytes of data.
.^C
--- 192.168.7.19 ping statistics ---
111404 packets transmitted, 111403 received, 0.000897634% packet loss, time 968ms
rtt min/avg/max/mdev = 0.091/0.106/3.053/0.018 ms, ipg/ewma 0.116/0.108 ms
root@pve9:~#

If I do a tail -f of the syslog, I see that messages are written but not always every second, there seems to be a small delay:

root@pve8:~# tail -f /var/log/syslog
Dec 29 04:00:25 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 679188
Dec 29 04:00:25 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 679189
Dec 29 04:00:25 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 67918a
Dec 29 04:00:25 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 67918e
Dec 29 04:00:30 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ab
Dec 29 04:00:30 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ac
Dec 29 04:00:30 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ad
Dec 29 04:00:30 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ae
Dec 29 04:00:30 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791af
Dec 29 04:00:30 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791b0
Dec 29 04:00:34 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791c8
Dec 29 04:00:34 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791c9
Dec 29 04:00:34 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ca
Dec 29 04:00:34 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791cb
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791e0
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791e1
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791e2
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791e7
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791e8
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791e9
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ed
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ee
Dec 29 04:00:35 pve8 corosync[2977]:   [TOTEM ] Retransmit List: 6791ef

Do you need any more information?

Thank you very much!!

 

[Christian St.]

Hello agapitox,

could you describe your corosync network in more detail? Is there just one link for corosync (is this link seperated from other traffic)? Are the other nodes on a different switch?
How are the switches connectet to each other?

 

[agapitox]

Hello agapitox,

could you describe your corosync network in more detail? Is there just one link for corosync (is this link seperated from other traffic)? Are the other nodes on a different switch?
How are the switches connectet to each other?

Hi Christian St.,

We have ten proxmox nodes distributed across different LAN switches; the ones that are showing the issue (pve8,pve10), are connected with one single gigabit ethernet link to the same LAN switch. There are 3 additional proxmox nodes in that same LAN switch (pve12, pve14, pve17) that are not showing the corosync errors. That LAN switch is connected to the network by two 10G uplinks. We don't see errors neither in the links from the proxmox nodes to the LAN switch nor in the 10G uplinks; and they have plenty of bandwidth.

On the other hand, there are no dedicated links for the corosync.

 

[Christian St.]

Corosync cares more about latency than bandwidth. In reference the corosync link should always be seperated from other network traffic.
Especially if the cluster becomes bigger, non seperated links could end up at a too high latency for corosync. It is also dangerus only having one link, because in the event of an single point failure (switch, interface, cable) the cluster can go down.

Even this page is for older versions, we normally test with omping. With knet only the unicast outcome is relevant. https://pve.proxmox.com/wiki/Multicast_notes

When using opming it should be started on all nodes of the cluster, which simulates that what corosync does.

omping -c 10000 -i 0.001 -F -q node1 node2 node3 nodeX (Using IP Adresses, if configured with ip-adresses)

Beware that, if you have poor-quality (or even just misconfigured) ethernet switches, some of these tests may crash your entire network, but at least then you'll know where the source of the problem is...

More information in the documentation: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_redundancy

Are these two nodes noticeable slower then other nodes in your cluster?
Are there VM/CT's on that nodes, that are using the network connection more intensive than on the other nodes?