[TOTEM ] FAILED TO RECEIVE

[Neustroev Andrei]

I have a 5 nodes in cluster. On 4 node i have error.

май 28 13:42:56 CCVM4 corosync[32562]: error   [TOTEM ] FAILED TO RECEIVE
май 28 13:42:56 CCVM4 corosync[32562]:  [TOTEM ] FAILED TO RECEIVE
май 28 13:42:59 CCVM4 corosync[32562]: notice  [TOTEM ] A new membership (192.168.211.23:287292) was formed. Members left: 4 1 2 3
май 28 13:42:59 CCVM4 corosync[32562]: notice  [TOTEM ] Failed to receive the leave message. failed: 4 1 2 3
май 28 13:42:59 CCVM4 corosync[32562]: warning [CPG   ] downlist left_list: 4 received
май 28 13:42:59 CCVM4 corosync[32562]: notice  [QUORUM] This node is within the non-primary component and will NOT provide any services.
май 28 13:42:59 CCVM4 corosync[32562]: notice  [QUORUM] Members[1]: 5
май 28 13:42:59 CCVM4 corosync[32562]: notice  [MAIN  ] Completed service synchronization, ready to provide service.
май 28 13:42:59 CCVM4 corosync[32562]:  [TOTEM ] A new membership (192.168.211.23:287292) was formed. Members left: 4 1 2 3
май 28 13:42:59 CCVM4 corosync[32562]:  [TOTEM ] Failed to receive the leave message. failed: 4 1 2 3
май 28 13:42:59 CCVM4 corosync[32562]:  [CPG   ] downlist left_list: 4 received
май 28 13:42:59 CCVM4 pmxcfs[32543]: [dcdb] notice: members: 5/32543
май 28 13:42:59 CCVM4 pmxcfs[32543]: [status] notice: members: 5/32543
май 28 13:42:59 CCVM4 corosync[32562]:  [QUORUM] This node is within the non-primary component and will NOT provide any services.
май 28 13:42:59 CCVM4 corosync[32562]:  [QUORUM] Members[1]: 5
май 28 13:42:59 CCVM4 corosync[32562]:  [MAIN  ] Completed service synchronization, ready to provide service.
май 28 13:42:59 CCVM4 pmxcfs[32543]: [status] notice: node lost quorum
май 28 13:42:59 CCVM4 pmxcfs[32543]: [dcdb] notice: cpg_send_message retried 1 times
май 28 13:42:59 CCVM4 pmxcfs[32543]: [dcdb] crit: received write while not quorate - trigger resync
май 28 13:42:59 CCVM4 pmxcfs[32543]: [dcdb] crit: leaving CPG group
май 28 13:43:00 CCVM4 pmxcfs[32543]: [dcdb] notice: start cluster connection
май 28 13:43:00 CCVM4 pmxcfs[32543]: [dcdb] notice: members: 5/32543
май 28 13:43:00 CCVM4 pmxcfs[32543]: [dcdb] notice: all data is up to date

Network configuration

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp7s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:25:90:4f:4c:e4 brd ff:ff:ff:ff:ff:ff
3: enp7s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:25:90:4f:4c:e4 brd ff:ff:ff:ff:ff:ff
4: enp7s0f2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:25:90:4f:4c:e4 brd ff:ff:ff:ff:ff:ff
5: enp7s0f3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 00:25:90:4f:4c:e4 brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr0 state UP group default qlen 1000
    link/ether 00:25:90:4f:4c:e4 brd ff:ff:ff:ff:ff:ff
7: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:25:90:4f:4c:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.211.23/23 brd 192.168.211.255 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::225:90ff:fe4f:4ce4/64 scope link
       valid_lft forever preferred_lft forever

/etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: CCVM1
    nodeid: 4
    quorum_votes: 1
    ring0_addr: CCVM1
  }
  node {
    name: CCVM2
    nodeid: 1
    quorum_votes: 1
    ring0_addr: CCVM2
  }
  node {
    name: CCVM3
    nodeid: 2
    quorum_votes: 1
    ring0_addr: CCVM3
  }
  node {
    name: CCVM4
    nodeid: 5
    quorum_votes: 1
    ring0_addr: CCVM4
  }
  node {
    name: CCVM5
    nodeid: 3
    quorum_votes: 1
    ring0_addr: CCVM5
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: CCVMS
  config_version: 13
  interface {
    bindnetaddr: 192.168.211.21
    ringnumber: 0
  }
  ip_version: ipv4
  secauth: on
  version: 2
}

 

[Neustroev Andrei]

omping 0% loss
iptables

iptables-save
# Generated by iptables-save v1.6.0 on Mon May 28 14:35:54 2018
*filter
:INPUT ACCEPT [6270331:106496523745]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [6860024:403984305]
COMMIT
# Completed on Mon May 28 14:35:54 2018