[SOLVED] Can't get the third node pi cluster working

[mac0s9user]

Hi Everyone,

I am trying to add a Raspberry Pi as a third node in an existing 2 node cluster. I had done this before and it worked fine but the SD card on my Pi died after a power outage. I couldnt get it to re-add so I rebuilt both of the proxmox servers as well as the Pi and I am still having no luck.

I follow the official Proxmox guide as well as several others and have had no luck. Here's what happens:

After adding the node it shows offline in the web GUI. After I reboot both hosts the Raspberry Pi totally vanishes in the GUI. The corosync file changes back to just a two node cluster like it was before.

The corosync-quorumtool on the pi shows activity blocked. This is the corosync file I pushed to the proxmox hosts

logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: skynetone
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.2.4
}
node {
name: skynettwo
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.2.44
}
node {
name: raspberrypi
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.2.6
}

}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: SkynetCluster
config_version: 2
interface {
linknumber: 0
}
ip_version: ipv4-6
secauth: on
version: 2
}

Maybe I am too far into the weeds at this point. The only issue I had was adding the gpg keys according to the guide but was able to pull corosync from the standard raspi sources list anyway.


corosync quorum on the pi says this:


Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Any ideas?

 

[Archmatux]

I would highly recommend not adding the raspberry pi as a third and equal node in the cluster.

I have had good success using a raspberry pi as a QDevice which can add a vote in the event of a tie.

See https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support for details on setting this up.

 

[mac0s9user]

Thank you! That's exactly what I want. I dont need it as a node but as a voting machine. I followed that guide but unfortunately the pi is still not getting a vote for some reason. This is off one of the nodes:

root@skynettwo:~# pvecm status
Quorum information
------------------
Date:             Sun Sep  1 10:28:54 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1/180
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2 
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 192.168.2.4
0x00000002          1   A,NV,NMW 192.168.2.44 (local)
0x00000000          0            Qdevice (votes 1)

 

[mac0s9user]

Yay! After a lot of banging my head on a wall and frustration I finally got this to work. Here is what I had to do:

On the pi:

remove corosync with

apt purge corosync

This probably wasn't necessary if I hadn't tried to do this in the past with trying to add it to the cluster.

Install both qnetd and qdevice with

apt install corosync-qnetd corosync-qdevice


For the Nodes, I had to do some things a little different then what was suggested by proxmox

Install both qnetd and qdevice with

apt install corosync-qnetd corosync-qdevice


Then edit corosync-qdevice file as there is a known bug that was giving me issues

========================================================
Bug info here: https://bugzilla.redhat.com/show_bug.cgi?id=1158805
========================================================
nano /etc/init.d/corosync-qdevice

In the header adjust it so it looks like this:

### BEGIN INIT INFO # Provides: corosync-qdevice # Required-Start: $remote_fs $syslog corosync # Required-Stop: $remote_fs $syslog corosync # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Corosync Qdevice daemon # Description: Starts and stops Corosync Qdevice daemon. ### END INIT INFO

You only have to do this on one node as it will be replicated out

Then:

update-rc.d -f corosync-qdevice remove

update-rc.d -f corosync-qdevice default

Finally you can run the following:

pvecm qdevice setup IPADDRESSOFPIHERE

On each node run the following

systemctl enable corosync-qdevice.service
systemctl start corosync-qdevice.service

Back on the pi run the following:
systemctl enable corosync-qnetd.service

systemctl start corosync-qnetd.service

Check your status on one of the nodes with

pvecm status

You should now see three devices checking in with one vote each.

 

[Ivan Dimitrov]

Ssuming you have only two nodes and wanting to run them in cluster. Alternative to adding raspberry Pi as 3rd node is to give pne of the nodes 2 votes. In such case you can be sure that always at least one of the nodes will be able to boot and you will be able to manage the cluster.

root@hyper-convergence:~# pvecm status
Quorum information
------------------
Date:             Sun Sep  1 23:28:51 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/52
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 10.10.0.3 (local)

 

[mac0s9user]

Ssuming you have only two nodes and wanting to run them in cluster. Alternative to adding raspberry Pi as 3rd node is to give pne of the nodes 2 votes. In such case you can be sure that always at least one of the nodes will be able to boot and you will be able to manage the cluster.

root@hyper-convergence:~# pvecm status
Quorum information
------------------
Date:             Sun Sep  1 23:28:51 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/52
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 10.10.0.3 (local)

That would also work. I actually got this resolved and have a lengthy write up in here. But it's awaiting moderator approval as I had to edit it to fix a typo so it's currently not visible