[SOLVED] Can't get the third node pi cluster working

mac0s9user

Member
Jun 9, 2019
4
0
21
38
Hi Everyone,

I am trying to add a Raspberry Pi as a third node in an existing 2 node cluster. I had done this before and it worked fine but the SD card on my Pi died after a power outage. I couldnt get it to re-add so I rebuilt both of the proxmox servers as well as the Pi and I am still having no luck.

I follow the official Proxmox guide as well as several others and have had no luck. Here's what happens:

After adding the node it shows offline in the web GUI. After I reboot both hosts the Raspberry Pi totally vanishes in the GUI. The corosync file changes back to just a two node cluster like it was before.

The corosync-quorumtool on the pi shows activity blocked. This is the corosync file I pushed to the proxmox hosts

logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: skynetone
nodeid: 1
quorum_votes: 1
ring0_addr: 192.168.2.4
}
node {
name: skynettwo
nodeid: 2
quorum_votes: 1
ring0_addr: 192.168.2.44
}
node {
name: raspberrypi
nodeid: 3
quorum_votes: 1
ring0_addr: 192.168.2.6
}

}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: SkynetCluster
config_version: 2
interface {
linknumber: 0
}
ip_version: ipv4-6
secauth: on
version: 2
}

Maybe I am too far into the weeds at this point. The only issue I had was adding the gpg keys according to the guide but was able to pull corosync from the standard raspi sources list anyway.


corosync quorum on the pi says this:


Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Any ideas?
 
Thank you! That's exactly what I want. I dont need it as a node but as a voting machine. I followed that guide but unfortunately the pi is still not getting a vote for some reason. This is off one of the nodes:

Code:
root@skynettwo:~# pvecm status
Quorum information
------------------
Date:             Sun Sep  1 10:28:54 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1/180
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2 
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 192.168.2.4
0x00000002          1   A,NV,NMW 192.168.2.44 (local)
0x00000000          0            Qdevice (votes 1)
 
Yay! After a lot of banging my head on a wall and frustration I finally got this to work. Here is what I had to do:

On the pi:

remove corosync with

apt purge corosync

This probably wasn't necessary if I hadn't tried to do this in the past with trying to add it to the cluster.

Install both qnetd and qdevice with

apt install corosync-qnetd corosync-qdevice


For the Nodes, I had to do some things a little different then what was suggested by proxmox

Install both qnetd and qdevice with

apt install corosync-qnetd corosync-qdevice


Then edit corosync-qdevice file as there is a known bug that was giving me issues

========================================================
Bug info here: https://bugzilla.redhat.com/show_bug.cgi?id=1158805
========================================================
nano /etc/init.d/corosync-qdevice

In the header adjust it so it looks like this:

### BEGIN INIT INFO # Provides: corosync-qdevice # Required-Start: $remote_fs $syslog corosync # Required-Stop: $remote_fs $syslog corosync # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Corosync Qdevice daemon # Description: Starts and stops Corosync Qdevice daemon. ### END INIT INFO

You only have to do this on one node as it will be replicated out

Then:

update-rc.d -f corosync-qdevice remove

update-rc.d -f corosync-qdevice default

Finally you can run the following:

pvecm qdevice setup IPADDRESSOFPIHERE

On each node run the following

systemctl enable corosync-qdevice.service
systemctl start corosync-qdevice.service

Back on the pi run the following:
systemctl enable corosync-qnetd.service

systemctl start corosync-qnetd.service

Check your status on one of the nodes with

pvecm status

You should now see three devices checking in with one vote each.
 
Last edited:
Ssuming you have only two nodes and wanting to run them in cluster. Alternative to adding raspberry Pi as 3rd node is to give pne of the nodes 2 votes. In such case you can be sure that always at least one of the nodes will be able to boot and you will be able to manage the cluster.

Code:
root@hyper-convergence:~# pvecm status
Quorum information
------------------
Date:             Sun Sep  1 23:28:51 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/52
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 10.10.0.3 (local)
 
  • Like
Reactions: mac0s9user
Ssuming you have only two nodes and wanting to run them in cluster. Alternative to adding raspberry Pi as 3rd node is to give pne of the nodes 2 votes. In such case you can be sure that always at least one of the nodes will be able to boot and you will be able to manage the cluster.

Code:
root@hyper-convergence:~# pvecm status
Quorum information
------------------
Date:             Sun Sep  1 23:28:51 2019
Quorum provider:  corosync_votequorum
Nodes:            1
Node ID:          0x00000001
Ring ID:          1/52
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 10.10.0.3 (local)


That would also work. I actually got this resolved and have a lengthy write up in here. But it's awaiting moderator approval as I had to edit it to fix a typo so it's currently not visible
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!