Qdevice

cpzengel · Nov 6, 2019

Hi,

i´d like to add a QDevice to my two machine Cluster.
I am failing in installing it on another single pve v6 because of the binary names have changed.
So I tried it in a Debian 9 Container, but its Still Corosync V2

What´s the best Practice? Any Documentation?

Cheers

Chriz

oguz · Nov 6, 2019

hi,

cpzengel said:
What´s the best Practice? Any Documentation?

best practice is to have a 2+1 setup, with 2 nodes in cluster and 1 qdevice separately monitoring it.

check the documentation[0].

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_qdevice_technical_overview

cpzengel · Nov 6, 2019

i´ve missed a package on node

is this ok so far?

Code:

Quorum information
------------------
Date:             Wed Nov  6 18:55:02 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.4e6dac
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2 
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 192.168.50.221 (local)
0x00000002          1         NR 192.168.50.223
0x00000000          0            Qdevice (votes 1)

Code:

Quorum information
------------------
Date:             Wed Nov  6 18:55:13 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1/5139884
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 192.168.50.221
0x00000002          1         NR 192.168.50.223 (local)
0x00000000          0            Qdevice (votes 1)
root@pve2:~#

oguz · Nov 6, 2019

cpzengel said:
i´ve missed a package on node

which package is missing?

cpzengel said:
is this ok so far?

your qdevice isn't voting. is it online?

cpzengel · Nov 6, 2019

Code:

/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
        (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Creating /etc/corosync/qnetd/nssdb
Creating new key and cert db
password file contains no data
Creating new noise file /etc/corosync/qnetd/nssdb/noise.txt
Creating new CA


Generating key.  This may take a few moments...

Is this a CA certificate [y/N]?
Enter the path length constraint, enter to skip [<0 for unlimited path]: > Is this a critical extension [y/N]?


Generating key.  This may take a few moments...

Notice: Trust flag u is set automatically if the private key is present.
QNetd CA certificate is exported as /etc/corosync/qnetd/nssdb/qnetd-cacert.crt

INFO: copying CA cert and initializing on all nodes

node 'pve1': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'pve1': Creating new key and cert db
node 'pve1': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pve1': Importing CA
node 'pve2': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'pve2': Creating new key and cert db
node 'pve2': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pve2': Importing CA
INFO: generating cert request
Creating new certificate request


Generating key.  This may take a few moments...

Certificate request stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.crq

INFO: copying exported cert request to qnetd server

INFO: sign and export cluster cert
Signing cluster certificate
Certificate stored in /etc/corosync/qnetd/nssdb/cluster-sysops-rz.crt

INFO: copy exported CRT

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes

node 'pve1': Importing cluster certificate and key
node 'pve1': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pve2': Importing cluster certificate and key
node 'pve2': pk12util: PKCS12 IMPORT SUCCESSFUL
INFO: add QDevice to cluster configuration

INFO: start and enable corosync qdevice daemon on node 'pve1'...
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
update-rc.d: error: corosync-qdevice Default-Start contains no runlevels, aborting.
command 'ssh -o 'BatchMode=yes' -lroot 192.168.50.221 systemctl enable corosync-qdevice' failed: exit code 1

cpzengel · Nov 6, 2019

https://forum.proxmox.com/threads/setting-up-qdevice-fails.56061/

now starting, still no vote

cpzengel · Nov 6, 2019

Nov 6 19:01:48 pve1 corosync-qdevice[58006]: Unhandled error when reading from server. Disconnecting from server

cpzengel · Nov 6, 2019

-- The process' exit code is 'exited' and its exit status is 1.

Nov 06 19:04:58 pve9 systemd[1]: corosync-qnetd.service: Failed with result 'exit-code'.

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- The unit corosync-qnetd.service has entered the 'failed' state with result 'exit-code'.

Nov 06 19:04:58 pve9 systemd[1]: Failed to start Corosync Qdevice Network daemon.

-- Subject: A start job for unit corosync-qnetd.service has failed

-- Defined-By: systemd

t.lamprecht · Nov 6, 2019

cpzengel said:
So I tried it in a Debian 9 Container, but its Still Corosync V2

Either use our stretch repo to pull our corosync, or use a Debian 10 CT.
The Debian 9 corosync packages have some issues regarding initscripts, IIRC.

t.lamprecht · Nov 6, 2019

Or maybe start the corosync-qdevice net daemon in the foreground to see if it's a init system issue or one from the daemon, it's config..

cpzengel · Nov 6, 2019

so its the autostart of the service thats failing
running in foreground is working!

cpzengel · Nov 6, 2019

its running on a pve6 non cluster, so i have to setup a own startscript?

t.lamprecht · Nov 6, 2019

cpzengel said:
its running on a pve6 non cluster, so i have to setup a own startscript?

Hmm, normally not.. Now, after some thinking I remember the external qdevice was split out for corosync 3, and we currently do not package it ourself (as it's separate thing and optional) so you'll get the one from Debian.. Need to re-check that package tomorrow (to late today sorry

)

cpzengel · Nov 6, 2019

One Node fine, the other that mess. Any Idea?

Code:

Quorum information
------------------
Date:             Wed Nov  6 23:22:41 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1/5139916
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 192.168.50.221
0x00000002          1         NR 192.168.50.223 (local)
0x00000000          0            Qdevice (votes 1)

cpzengel · Nov 6, 2019

after removing and adding qdevice for the moment ok

TiagoLardaCrianca · Sep 9, 2022

for me it had to do with firewall blocking packets

I used sudo ufw allow 5403/tcp and it added the vote i needed

pieteras.meyer · Oct 11, 2022

Hi,
Does a QDevice work fine with a 6node Proxmox VM cluster? all nodes is on V7.2

or is the below still a limitations from the manual? Or is the below only true for Uneven number clusters?

If the QNet daemon itself fails, no other node may fail or the cluster immediately loses quorum. For example, in a cluster with 15 nodes, 7 could fail before the cluster becomes inquorate. But, if a QDevice is configured here and it itself fails, no single node of the 15 may fail. The QDevice acts almost as a single point of failure in this case.

t.lamprecht · Oct 11, 2022

pieteras.meyer said:
Does a QDevice work fine with a 6node Proxmox VM cluster? all nodes is on V7.2

Yes.

pieteras.meyer said:
or is the below still a limitations from the manual? Or is the below only true for Uneven number clusters?

The "Supported Setups" section starts with the IMO pretty clear:

We support QDevices for clusters with an even number of nodes and recommend it for 2 node clusters, if they should provide higher availability. For clusters with an odd node count, we currently discourage the use of QDevices.

Following with reasoning and context for why we discourage using it on clusters with odd-numbered node counts.

pieteras.meyer · Oct 11, 2022

Hi Thomas

Will it allow 3 nodes being down?
3 in one rack of a Datacentre and 3 in another Rack in a another Datacentre,

The 2 racks are connected via Fibre,

So will a Qdevice allow 1 Datacentre to go offine and still survive ?

t.lamprecht · Oct 11, 2022

pieteras.meyer said:
Will it allow 3 nodes being down?

pieteras.meyer said:
So will a Qdevice allow 1 Datacentre to go offine and still survive ?

As also described in the docs: yes, as long as the QDevice is hosted somewhere independent of the Proxmox VE cluster it gives a vote too.

But note the following QDevice independent points that are very important:

the storage that the virtual guests are using needs to be able to handle this too.
Ceph, for example has its own cluster quorum provided by the Ceph monitors, not Proxmox VE. So there you would need something like a monitor on each datacenter site and an external monitor for tie breaking.
you need to have enough memory and CPU resources to keep your whole workload running on only three nodes.

Qdevice

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Retired Staff

Renowned Member

Renowned Member

Renowned Member

Renowned Member

Proxmox Staff Member

Proxmox Staff Member

Renowned Member

Renowned Member

Proxmox Staff Member

Renowned Member

Renowned Member

Member

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member