Update TOTEM Error (Cluster Error)

[Realtox]

Hello Guys,

i have Proxmox 7.2-11 with 8 Nodes.

On the newest Update, there is a weard error.

Sometimes the login doesn't work because the PVE daemon doesn't respond.

I become this SysLog:

Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] link: host: 6 link: 0 is down
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] link: host: 6 link: 1 is down
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] link: host: 2 link: 0 is down
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] host: host: 6 has no active links
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] host: host: 6 has no active links
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 24 20:59:36 pve2 corosync[1459]:   [KNET  ] host: host: 2 has no active links
Oct 24 20:59:37 pve2 sshd[775314]: Failed password for root from 61.177.173.49 port 62191 ssh2
Oct 24 20:59:38 pve2 sshd[775312]: Failed password for root from 61.177.172.143 port 58510 ssh2
Oct 24 20:59:39 pve2 corosync[1459]:   [KNET  ] rx: host: 6 link: 0 is up
Oct 24 20:59:39 pve2 corosync[1459]:   [KNET  ] rx: host: 2 link: 0 is up
Oct 24 20:59:39 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:39 pve2 corosync[1459]:   [KNET  ] rx: host: 6 link: 1 is up
Oct 24 20:59:39 pve2 corosync[1459]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 24 20:59:39 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:42 pve2 sshd[775314]: Failed password for root from 61.177.173.49 port 62191 ssh2
Oct 24 20:59:43 pve2 corosync[1459]:   [KNET  ] link: host: 7 link: 0 is down
Oct 24 20:59:43 pve2 corosync[1459]:   [KNET  ] host: host: 7 (passive) best link: 0 (pri: 1)
Oct 24 20:59:43 pve2 corosync[1459]:   [KNET  ] host: host: 7 has no active links
Oct 24 20:59:45 pve2 sshd[775314]: Failed password for root from 61.177.173.49 port 62191 ssh2
Oct 24 20:59:45 pve2 corosync[1459]:   [QUORUM] Sync members[7]: 1 2 4 5 6 7 8
Oct 24 20:59:45 pve2 corosync[1459]:   [TOTEM ] A new membership (1.2d4c7) was formed. Members
Oct 24 20:59:46 pve2 corosync[1459]:   [KNET  ] link: host: 6 link: 0 is down
Oct 24 20:59:46 pve2 corosync[1459]:   [KNET  ] link: host: 6 link: 1 is down
Oct 24 20:59:46 pve2 corosync[1459]:   [KNET  ] link: host: 2 link: 0 is down
Oct 24 20:59:47 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:47 pve2 corosync[1459]:   [KNET  ] host: host: 6 has no active links
Oct 24 20:59:47 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:47 pve2 corosync[1459]:   [KNET  ] host: host: 6 has no active links
Oct 24 20:59:47 pve2 corosync[1459]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 24 20:59:47 pve2 corosync[1459]:   [KNET  ] host: host: 2 has no active links
Oct 24 20:59:48 pve2 pmxcfs[1376]: [status] notice: cpg_send_message retry 10
Oct 24 20:59:48 pve2 corosync[1459]:   [KNET  ] rx: host: 7 link: 0 is up
Oct 24 20:59:48 pve2 corosync[1459]:   [KNET  ] host: host: 7 (passive) best link: 0 (pri: 1)
Oct 24 20:59:49 pve2 pmxcfs[1376]: [status] notice: cpg_send_message retry 20
Oct 24 20:59:50 pve2 pmxcfs[1376]: [status] notice: cpg_send_message retry 30
Oct 24 20:59:50 pve2 corosync[1459]:   [KNET  ] rx: host: 6 link: 1 is up
Oct 24 20:59:50 pve2 corosync[1459]:   [KNET  ] rx: host: 6 link: 0 is up
Oct 24 20:59:50 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:50 pve2 corosync[1459]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Oct 24 20:59:50 pve2 corosync[1459]:   [KNET  ] rx: host: 2 link: 0 is up
Oct 24 20:59:50 pve2 corosync[1459]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Oct 24 20:59:51 pve2 pmxcfs[1376]: [status] notice: cpg_send_message retry 40
Oct 24 20:59:52 pve2 pmxcfs[1376]: [status] notice: cpg_send_message retry 50
Oct 24 20:59:52 pve2 corosync[1459]:   [QUORUM] Sync members[7]: 1 2 4 5 6 7 8
Oct 24 20:59:52 pve2 corosync[1459]:   [TOTEM ] A new membership (1.2d4cb) was formed. Members
Oct 24 20:59:52 pve2 corosync[1459]:   [TOTEM ] Retransmit List: 3f
Oct 24 20:59:53 pve2 corosync[1459]:   [QUORUM] Members[7]: 1 2 4 5 6 7 8
Oct 24 20:59:53 pve2 corosync[1459]:   [MAIN  ] Completed service synchronization, ready to provide service.
Oct 24 20:59:53 pve2 pmxcfs[1376]: [status] notice: cpg_send_message retried 57 times
Oct 24 20:59:54 pve2 pvestatd[1480]: status update time (7.021 seconds)
Oct 24 20:59:58 pve2 pve-ha-lrm[1566]: loop take too long (33 seconds)

Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/176: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/125: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/166: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/168: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/1003: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/163: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/150: -1
Oct 24 20:59:17 pve2 pmxcfs[1376]: [status] notice: RRDC update error /var/lib/rrdcached/db/pve2-vm/113: -1

Can anybody help or has the same error?

Kind Regards
Felix

 

[mira]

The 2nd log looks like your time is not synchronized. Make sure that it is!
Do you have `chrony` installed or do you use `systemd-timesyncd`? If the latter, please install chrony since it is a lot more reliable.