'pvecm qdevice setup' fails

joachim

Member
Nov 19, 2015
16
0
21
Hi,

Trying to set up QDevice for a 2-node PVE cluster.

I've installed corosync-qnetd on a Raspberry Pi, and corosync-qdevice on both PVE nodes.

When trying to run the configuration from one of the PVE nodes, it fails;

Code:
root@gridlock:~# pvecm qdevice setup 2001:123:123:123::123
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/usr/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
        (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes
Certificate database already exists. Delete it to continue
Host key verification failed.

INFO: generating cert request
Certificate database doesn't exists. Use /usr/sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n pve-cluster1' failed: exit code 1


The install log from QDevice on the Raspberry Pi;

Code:
Preparing to unpack .../corosync-qnetd_3.0.0-4+deb10u1_armhf.deb ...
Unpacking corosync-qnetd (3.0.0-4+deb10u1) ...
Setting up corosync-qnetd (3.0.0-4+deb10u1) ...
Creating /etc/corosync/qnetd/nssdb
Creating new key and cert db
password file contains no data
Creating new noise file /etc/corosync/qnetd/nssdb/noise.txt
Creating new CA


Generating key.  This may take a few moments...

Is this a CA certificate [y/N]?
Enter the path length constraint, enter to skip [<0 for unlimited path]: > Is this a critical extension [y/N]?


Generating key.  This may take a few moments...

Notice: Trust flag u is set automatically if the private key is present.
QNetd CA certificate is exported as /etc/corosync/qnetd/nssdb/qnetd-cacert.crt
Created symlink /etc/systemd/system/multi-user.target.wants/corosync-qnetd.service → /lib/systemd/system/corosync-qnetd.service.
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for systemd (241-7~deb10u5+rpi1) ...

qnetd service running on the Raspberry Pi:

Code:
root@gumpii:~# systemctl status corosync-qnetd
● corosync-qnetd.service - Corosync Qdevice Network daemon
   Loaded: loaded (/lib/systemd/system/corosync-qnetd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-05-05 02:13:35 CEST; 32min ago
     Docs: man:corosync-qnetd
 Main PID: 22326 (corosync-qnetd)
    Tasks: 1 (limit: 2063)
   CGroup: /system.slice/corosync-qnetd.service
           └─22326 /usr/bin/corosync-qnetd -f

May 05 02:13:35 gumpii systemd[1]: Starting Corosync Qdevice Network daemon...
May 05 02:13:35 gumpii systemd[1]: Started Corosync Qdevice Network daemon.
 
Last edited:

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
3,276
373
88
hi,

what do you see from pvecm status?

if you see the qdevice but it doesn't have a vote, please try following:
Code:
pvecm qdevice remove
pvecm qdevice setup <IP> --force

and then check cluster status again with pvecm status
 

joachim

Member
Nov 19, 2015
16
0
21
hi,

what do you see from pvecm status?

if you see the qdevice but it doesn't have a vote, please try following:
Code:
pvecm qdevice remove
pvecm qdevice setup <IP> --force

and then check cluster status again with pvecm status

Hi,

No QDevice is configured;

Code:
root@gridlock:~# pvecm status
Cluster information
-------------------
Name:             pve-cluster1
Config Version:   2
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Wed May  5 13:14:43 2021
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.9
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 2000:123:123:123::10%32635 (local)
0x00000002          1 2000:123:123:123::11%32635
root@gridlock:~# pvecm qdevice remove
error during cfs-locked 'file-corosync_conf' operation: No QDevice configured!
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
3,276
373
88
have you enabled PermitRootLogin yes on your /etc/ssh/sshd_config in the Pi and restarted sshd?

you should be able to ssh from your node to there with a password, which you type when you're setting up the qdevice via pvecm qdevice setup <IP>
 

joachim

Member
Nov 19, 2015
16
0
21
have you enabled PermitRootLogin yes on your /etc/ssh/sshd_config in the Pi and restarted sshd?

you should be able to ssh from your node to there with a password, which you type when you're setting up the qdevice via pvecm qdevice setup <IP>

Yes, that works just fine. If you look at the logs in my first post, you can see that ssh-copy-id "complains" that the SSH-key already exists on the target system (impling that it a) can log in, and b) the SSH-key is already present). Doing a manual login using password also still works.
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
3,276
373
88
and is there a key there already from before maybe?
 

joachim

Member
Nov 19, 2015
16
0
21
and is there a key there already from before maybe?

Yes, there are multiple keys there (for other logins). Not sure how that is relevant? The SSH-key is present, and SSH-key based login from the two PVE nodes works just fine...
 

oguz

Proxmox Staff Member
Staff member
Nov 19, 2018
3,276
373
88
please remove all the keys from there and try to run the setup again. you can add your keys afterwards
 

joachim

Member
Nov 19, 2015
16
0
21
please remove all the keys from there and try to run the setup again. you can add your keys afterwards
Tried that, but still the same;


Code:
root@gridlock:~# pvecm qdevice setup 2000:123:123:123::210
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@2001:67c:197c:110::210's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@2000:123:123:123::210'"
and check to make sure that only the key(s) you wanted were added.


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes
Certificate database already exists. Delete it to continue
Host key verification failed.

INFO: generating cert request
Certificate database doesn't exists. Use /usr/sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n pve-cluster1' failed: exit code 1
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!