quorum device setup issue "corosync-qdevice Default-Start contains no runlevels"

Giovanni

Renowned Member
Apr 1, 2009
111
11
83
Hi,

I'm having some trouble setting up the qdevice following the documentation [1]

Code:
# pvecm qdevice setup 172.16.0.69 -f
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes

node 'VE-H310': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'VE-H310': Creating new key and cert db
node 'VE-H310': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'VE-H310': Importing CA
node 'VE-SSD': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'VE-SSD': Creating new key and cert db
node 'VE-SSD': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'VE-SSD': Importing CA
INFO: generating cert request
Creating new certificate request


Generating key.  This may take a few moments...

Certificate request stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.crq

INFO: copying exported cert request to qnetd server

INFO: sign and export cluster cert
Signing cluster certificate
Certificate stored in /etc/corosync/qnetd/nssdb/cluster-LAX.crt

INFO: copy exported CRT

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes

node 'VE-H310': Importing cluster certificate and key
node 'VE-H310': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'VE-SSD': Importing cluster certificate and key
node 'VE-SSD': pk12util: PKCS12 IMPORT SUCCESSFUL
INFO: add QDevice to cluster configuration

INFO: start and enable corosync qdevice daemon on node 'VE-H310'...
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
update-rc.d: error: corosync-qdevice Default-Start contains no runlevels, aborting.
command 'ssh -o 'BatchMode=yes' -lroot 192.168.255.151 systemctl enable corosync-qdevice' failed: exit code 1
root@VE-SSD:~#

I've logged onto the PVE nodes and tried to run the systemctl enable command and both of them fail. Not sure if there's a new requirement or something I missed.


[1] https://pve.proxmox.com/wiki/Cluster_Manager
 
On the PVE nodes I edited the startup script and added some info to the runlevels. Command now works but forcing a re-setup fails, seems like it thinks the qdevice was setup properly and must be removed.

/etc/init.d/corosync-qdevice

### BEGIN INIT INFO
# Provides: corosync-qdevice
# Required-Start: $remote_fs $syslog corosync
# Required-Stop: $remote_fs $syslog corosync
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Corosync Qdevice daemon
# Description: Starts and stops Corosync Qdevice daemon.
### END INIT INFO

Hmmm....

Code:
root@VE-SSD:~# pvecm qdevice remove
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable corosync-qdevice
update-rc.d: error: no runlevel symlinks to modify, aborting!
command 'ssh -o 'BatchMode=yes' -lroot 192.168.255.151 systemctl disable corosync-qdevice' failed: exit code 1
 
D'oh forgot to regen the init.d scripts after the edit this will do:

# update-rc.d -f corosync-qdevice remove
# update-rc.d -f corosync-qdevice defaults

After that I remove the qdevice settings
# pvecm qdevice remove

and re-try
Code:
# pvecm qdevice setup 172.16.0.69 -f
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes

node 'VE-H310': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'VE-H310': Creating new key and cert db
node 'VE-H310': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'VE-H310': Importing CA
node 'VE-SSD': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'VE-SSD': Creating new key and cert db
node 'VE-SSD': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'VE-SSD': Importing CA
INFO: generating cert request
Creating new certificate request


Generating key.  This may take a few moments...

Certificate request stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.crq

INFO: copying exported cert request to qnetd server

INFO: sign and export cluster cert
Signing cluster certificate
Certificate stored in /etc/corosync/qnetd/nssdb/cluster-LAX.crt

INFO: copy exported CRT

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes

node 'VE-H310': Importing cluster certificate and key
node 'VE-H310': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'VE-SSD': Importing cluster certificate and key
node 'VE-SSD': pk12util: PKCS12 IMPORT SUCCESSFUL
INFO: add QDevice to cluster configuration

INFO: start and enable corosync qdevice daemon on node 'VE-H310'...
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink /etc/systemd/system/multi-user.target.wants/corosync-qdevice.service -> /lib/systemd/system/corosync-qdevice.service.

INFO: start and enable corosync qdevice daemon on node 'VE-SSD'...
Synchronizing state of corosync-qdevice.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink /etc/systemd/system/multi-user.target.wants/corosync-qdevice.service -> /lib/systemd/system/corosync-qdevice.service.
Reloading corosync.conf...
Done
root@VE-SSD:~# pvecm status
Quorum information
------------------
Date:             Wed Oct 30 01:36:29 2019
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000002
Ring ID:          1/16
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 192.168.255.151
0x00000002          1    A,V,NMW 192.168.255.150 (local)
0x00000000          1            Qdevice
root@VE-SSD:~#

I would recommend we add a note to documentation about this issue, I was able to find a fix in [1] https://bugs.launchpad.net/ubuntu/+source/corosync-qdevice/+bug/1809682
 
  • Like
Reactions: idic

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!