Quorum Activity blocked

[djahida]

Hi,
I have trouble on my cluster pve 4.3 with 3 nodes.
Two nodes are seeing each other, but the third doesn’t see the two others nodes.
Some advice, please, how to resolve this problem.


1. /etc/pve/corosync is readonly on the node 2.
2. pvecm s on node 1 and 3
root@node3 : ~ # pvecm s
Quorum information
------------------
Date: Sat Jun 24 14:35:43 2017
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000003
Ring ID: 3/2004652
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000003 1 node3 (local)
0x00000001 1 node1

3. pvecm s on node 2
root@node 2: ~ # pvecm s
Quorum information
------------------
Date: Sat Jun 24 15:16:03 2017
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000002
Ring ID: 2/1841220
Quorate: No

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000002 1 node 2 (local)


4. Log on node 1
root@node1: /etc # tail -f /var/log/syslog
J
Jun 24 15:08:44 node1corosync[3107]: [MAIN ] Completed service synchronization, ready to provide service.
Jun 24 15:08:47 node1corosync[3107]: [TOTEM ] A new membership (node3:2008392) was formed. Members
Jun 24 15:08:47 node1corosync[3107]: [QUORUM] Members[2]: 3 1
Jun 24 15:08:47 node1corosync[3107]: [MAIN ] Completed service synchronization, ready to provide service.
Jun 24 15:08:49 node1corosync[3107]: [TOTEM ] A new membership (node3:2008396) was formed. Members
Jun 24 15:08:49 node1corosync[3107]: [QUORUM] Members[2]: 3 1

5. Log on node 2
Jun 24 14:35:11 node2 pve-ha-lrm[3225]: ipcc_send_rec failed: Transport endpoint is not connected
Jun 24 14:35:11 node2 pve-ha-crm[3213]: ipcc_send_rec failed: Transport endpoint is not connected
Jun 24 14:35:11 node2 PP007 pve-ha-crm[3213]: ipcc_send_rec failed: Connection refused
Jun 24 14:35:11 node2 pve-ha-lrm[3225]: ipcc_send_rec failed: Connection refused
Jun 24 14:35:11 node2 pve-ha-crm[3213]: ipcc_send_rec failed: Connection refused
Jun 24 14:35:11 node2 pve-ha-lrm[3225]: ipcc_send_rec failed: Connection refused
Jun 24 14:35:12 node2 pmxcfs[4909]: [main] notice: exit proxmox configuration filesystem (0)
Jun 24 14:35:12 node2 systemd[1]: Starting The Proxmox VE cluster filesystem...
Jun 24 14:35:12 node2 pveproxy[4352]: ipcc_send_rec failed: Transport endpoint is not connected
Jun 24 14:35:12 node2 pveproxy[4352]: ipcc_send_rec failed: Connection refused
Jun 24 14:35:12 node2 pveproxy[4352]: ipcc_send_rec failed: Connection refused

6. Log on node 3
root@node3 : ~ # tail -f /var/log/syslog
Jun 24 15:10:52 node3 corosync[18114]: [MAIN ] Completed service synchronization, ready to provide service.
Jun 24 15:10:54 node3 corosync[18114]: [TOTEM ] A new membership (node3:2008632) was formed. Members
Jun 24 15:10:54 node3 corosync[18114]: [QUORUM] Members[2]: 3 1
Jun 24 15:10:54 node3 corosync[18114]: [MAIN ] Completed service synchronization, ready to provide service.
Jun 24 15:10:56 node3 corosync[18114]: [TOTEM ] A new membership (node3 :2008636) was formed. Members
Jun 24 15:10:56 node3 corosync[18114]: [QUORUM] Members[2]: 3 1

Thank's

 

[dietmar]

Seems pmxcfs was stopped by someone (on node2)? Or what happened on node2 by that time? I would reboot to resolve that cleanly.

 

[promok]

Hello,
I have similar problem on my simple setup. I want to build a cluster and i cannot. I have only 1 network and the latest proxmox install.
=============================================================================
NODE1: (worked OK, i have set up a new cluster on it)
Cluster information
-------------------
Name: dellcluster
Config Version: 6
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Fri Jul 8 18:48:37 2022
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000001
Ring ID: 1.a
Quorate: No

Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 192.168.1.40 (local)
-----------------------------------------------------------------------------------------------------
NODE2: (which cannot join the cluster and gets broken)
Cluster information
-------------------
Name: dellcluster
Config Version: 6
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Fri Jul 8 18:48:14 2022
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000002
Ring ID: 2.a
Quorate: No

Votequorum information
----------------------
Expected votes: 2
Highest expected: 2
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000002 1 192.168.1.30 (local)
=============================================
On node2 the syslog looks like:
Jul 8 19:16:38 proxmox2 pveproxy[1414]: starting 1 worker(s)
Jul 8 19:16:38 proxmox2 pveproxy[1414]: worker 12737 started
Jul 8 19:16:38 proxmox2 pveproxy[12736]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/API
Server/AnyEvent.pm line 1943.
Jul 8 19:16:38 proxmox2 pveproxy[12737]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/API
Server/AnyEvent.pm line 1943.
Jul 8 19:16:38 proxmox2 pveproxy[1414]: worker 12732 finished
Jul 8 19:16:38 proxmox2 pveproxy[1414]: starting 1 worker(s)
Jul 8 19:16:38 proxmox2 pveproxy[1414]: worker 12738 started
Jul 8 19:16:38 proxmox2 pveproxy[12738]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/API
Server/AnyEvent.pm line 1943.
-------------------------------------------------
what key? why? on new setup from scratch?
root@proxmox2:/etc/pve# ls -la
total 5
drwxr-xr-x 2 root www-data 0 Jan 1 1970 .
drwxr-xr-x 90 root root 4096 Jul 8 17:06 ..
-r--r----- 1 root www-data 591 Jan 1 1970 .clusterlog
-r--r----- 1 root www-data 455 Jul 8 18:02 corosync.conf
-rw-r----- 1 root www-data 2 Jan 1 1970 .debug
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 local -> nodes/proxmox2
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 lxc -> nodes/proxmox2/lxc
-r--r----- 1 root www-data 237 Jan 1 1970 .members
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 openvz -> nodes/proxmox2/openvz
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 qemu-server -> nodes/proxmox2/qemu-server
-r--r----- 1 root www-data 222 Jan 1 1970 .rrd
-r--r----- 1 root www-data 710 Jan 1 1970 .version
-r--r----- 1 root www-data 18 Jan 1 1970 .vmlist
------------
i see that the destinations for these 4 links do NOT exist!!!! There are NO subdirectories in /etc/pve on the second node which i wanted to add to the cluster.
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 local -> nodes/proxmox2
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 lxc -> nodes/proxmox2/lxc
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 openvz -> nodes/proxmox2/openvz
lr-xr-xr-x 1 root www-data 0 Jan 1 1970 qemu-server -> nodes/proxmox2/qemu-server

if this is not normal, then this is the proxmox code which is broken on proxmox 7.2.1.
Could you fix it, please?
what to do next?
Thanks.