2 Node Cluster | Advice on Adding QDevice

[gravyflex]

Hello,

I'm running a two-node proxmox v7.0-11 cluster. I'm adding a raspberry pi as a qdevice.

The proxmox host are running qdevice version

# apt list --installed | grep corosync-qdevice
corosync-qdevice/stable,now 3.0.1-1 amd64 [installed]

The pi is running Raspbian GNU/Linux 10 (buster)

apt list --installed | grep corosync-qdevice
corosync-qdevice/oldstable,now 3.0.0-4+deb10u1 armhf [installed]

I'm trying to establish if the difference in corosync-qdevice versions between the proxmox hosts and raspberry pi would make a difference.

I would be grateful for some guidance.

 

[itNGO]

Hi,
in theory v3.xxx Corosync is fully compatible. However, keep an sharp eye on the logs when adding the 3rd "node".

 

[gravyflex]

I've added the qDevice

INFO: start and enable corosync qdevice daemon on node 'drogon01'...
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink /etc/systemd/system/multi-user.target.wants/corosync-qdevice.service -> /lib/systemd/system/corosync-qdevice.service.

INFO: start and enable corosync qdevice daemon on node 'drogon02'...
Job for corosync-qdevice.service failed because the control process exited with error code.
See "systemctl status corosync-qdevice.service" and "journalctl -xe" for details.
command 'ssh -o 'BatchMode=yes' -lroot 10.168.20.142 systemctl start corosync-qdevice' failed: exit code 1

On node2
I am seeing this error

journalctl --since "1 hour ago" -u  corosync-qdevice
-- Journal begins at Tue 2021-08-31 18:07:08 CDT, ends at Thu 2021-11-04 17:13:00 CDT. --
Nov 04 17:01:20 drogon02 systemd[1]: Starting Corosync Qdevice daemon...
Nov 04 17:01:20 drogon02 corosync-qdevice[2047195]: Can't read quorum.device.model cmap key.
Nov 04 17:01:20 drogon02 systemd[1]: corosync-qdevice.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 17:01:20 drogon02 systemd[1]: corosync-qdevice.service: Failed with result 'exit-code'.
Nov 04 17:01:20 drogon02 systemd[1]: Failed to start Corosync Qdevice daemon.
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Scheduled restart job, restart counter is at 1.
Nov 04 17:01:21 drogon02 systemd[1]: Stopped Corosync Qdevice daemon.
Nov 04 17:01:21 drogon02 systemd[1]: Starting Corosync Qdevice daemon...
Nov 04 17:01:21 drogon02 corosync-qdevice[2047199]: Can't read quorum.device.model cmap key.
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Failed with result 'exit-code'.
Nov 04 17:01:21 drogon02 systemd[1]: Failed to start Corosync Qdevice daemon.
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Scheduled restart job, restart counter is at 2.
Nov 04 17:01:21 drogon02 systemd[1]: Stopped Corosync Qdevice daemon.
Nov 04 17:01:21 drogon02 systemd[1]: Starting Corosync Qdevice daemon...
Nov 04 17:01:21 drogon02 corosync-qdevice[2047204]: Can't read quorum.device.model cmap key.
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Main process exited, code=exited, status=1/FAILURE
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Failed with result 'exit-code'.
Nov 04 17:01:21 drogon02 systemd[1]: Failed to start Corosync Qdevice daemon.
Nov 04 17:01:21 drogon02 systemd[1]: corosync-qdevice.service: Scheduled restart job, restart counter is at 3.
Nov 04 17:01:21 drogon02 systemd[1]: Stopped Corosync Qdevice daemon.
Nov 04 17:01:21 drogon02 systemd[1]: Starting Corosync Qdevice daemon...
Nov 04 17:01:21 drogon02 corosync-qdevice[2047208]: Can't read quorum.device.model cmap key.

node 1 pvecm status

pvecm status
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_TERMINAL = "iTerm2",
    LC_CTYPE = "en_CA.UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
Cluster information
-------------------
Name:             drogon
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Nov  4 17:15:57 2021
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.11e
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 10.168.20.141 (local)
0x00000002          1  NA,NV,NMW 10.168.20.142
0x00000000          1            Qdevice

node2

 pvecm status
Cluster information
-------------------
Name:             drogon
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Thu Nov  4 17:17:32 2021
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1.11e
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 10.168.20.141
0x00000002          1  NA,NV,NMW 10.168.20.142 (local)
0x00000000          0            Qdevice (votes 1)

I am digging into how to resolve it. I don't want to do something dumb. Any ideas?

 

[itNGO]

Hi,
maybe you can get some hints from this template.... https://florianmuller.com/setup-a-p...berrypi-as-a-third-quorum-vote-device-qdevice or https://www.danatec.org/2021/05/21/two-node-cluster-in-proxmox-ve-with-raspberry-pi-as-qdevice/

 

[gravyflex]

Thank you for your response. I had used that blog post as a part of my research.

I was able to resolve the issue by restarting the second node.