[SOLVED] Trying to add a QDevice

P

puntoboy

New Member
Apr 11, 2024
9
2
3
I've been trying for a few days now to add a QDevice to my PVE setup. I had a single node, and wanted to add a second with ZFR replication. Because of that I wanted to add a QDevice as a third vote.

I Created the PVE cluster, that was all fine but I'm stuck at adding the QDevice. I was getting "Host key verification failed." but after reading a few posted I manually copied the file from the node to the second node. This cleared that error but I still don't think the QDevice is actually doing anything.

The Contents of pvecm status on node1 is below

Code: 
root@pve:~# pvecm status
Cluster information
-------------------
Name:             pvc
Config Version:   8
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Sun May  5 17:19:06 2024
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.311
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 192.168.1.108 (local)
0x00000002          1 192.168.1.109

So I only see 2 votes required and I'm expecting to see 3.

This is what I ran on the QDevice and it's output now.

Code: 
root@pve:~# pvecm qdevice setup 192.168.1.150 --force
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes
bash: line 1: corosync-qdevice-net-certutil: command not found

node 'pve': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'pve': Creating new key and cert db
node 'pve': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pve': Importing CA
INFO: generating cert request
Creating new certificate request


Generating key.  This may take a few moments...

Certificate request stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.crq

INFO: copying exported cert request to qnetd server

INFO: sign and export cluster cert
Signing cluster certificate
Certificate stored in /etc/corosync/qnetd/nssdb/cluster-pvc.crt

INFO: copy exported CRT

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes
bash: line 1: corosync-qdevice-net-certutil: command not found
command 'ssh -o 'BatchMode=yes' -lroot 192.168.1.109 corosync-qdevice-net-certutil -m -c /etc/pve/qdevice-net-node.p12' failed: exit code 127
 
I've gone through this a gazillion times today. It keeps failing for me also.

I keep getting key verification failed although I've regenerated the certs and I can SSH between nodes and into 192.168.0.88 just using ssh@ip

Code: 
root@proxR86S:~# pvecm qdevice setup 192.168.0.88 --force
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed


/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)




INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db


INFO: copying CA cert and initializing on all nodes
Host key verification failed.
Certificate database already exists. Delete it to continue


INFO: generating cert request
Certificate database doesn't exists. Use /sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n ProxCluster1' failed: exit code 1

Have corosync-qdevice installed on all nodes.
 
Last edited:
Weidah said:
I've gone through this a gazillion times today. It keeps failing for me also.

I keep getting key verification failed although I've regenerated the certs and I can SSH between nodes and into 192.168.0.88 just using ssh@ip

Code: 
root@proxR86S:~# pvecm qdevice setup 192.168.0.88 --force
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed


/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)




INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db


INFO: copying CA cert and initializing on all nodes
Host key verification failed.
Certificate database already exists. Delete it to continue


INFO: generating cert request
Certificate database doesn't exists. Use /sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n ProxCluster1' failed: exit code 1

Have corosync-qdevice installed on all nodes.
Click to expand...

Simply ssh from between all nodes and qdevice and accept everytime once with "yes"
Its stupid, but it fails because of the first time ssh dialog, where it asks if "you are sure want to connect" or accept the key: yes/no dialog.
So you have to ssh connect once between all nodes and retry the setup then.
 
Ramalama said:
Simply ssh from between all nodes and qdevice and accept everytime once with "yes"
Its stupid, but it fails because of the first time ssh dialog, where it asks if "you are sure want to connect" or accept the key: yes/no dialog.
So you have to ssh connect once between all nodes and retry the setup then.
Click to expand...

Issue resolved, see above. :) thanks
 
  • Like
Reactions: Ramalama
Ramalama said:
Simply ssh from between all nodes and qdevice and accept everytime once with "yes"
Its stupid, but it fails because of the first time ssh dialog, where it asks if "you are sure want to connect" or accept the key: yes/no dialog.
So you have to ssh connect once between all nodes and retry the setup then.
Click to expand...
Yeah no, I thought of that and did that already also if you check my post again :)


But what I did now is I went and ssh-keygen -R on all nodes and qdevice for all of them. Basically went on each and ran the command for each IP node and qdevice.

I then went and SSH-ed from each device to each other device
I then deleted all the nssdb folders on nodes and qdevice
I then reran the command without --force



Code: 
root@proxR86S:~# pvecm qdevice setup 192.168.0.88
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed


/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)




INFO: initializing qnetd server
Creating /etc/corosync/qnetd/nssdb
Creating new key and cert db
password file contains no data
Creating new noise file /etc/corosync/qnetd/nssdb/noise.txt
Creating new CA




Generating key.  This may take a few moments...


Is this a CA certificate [y/N]?
Enter the path length constraint, enter to skip [<0 for unlimited path]: > Is this a critical extension [y/N]?




Generating key.  This may take a few moments...


Notice: Trust flag u is set automatically if the private key is present.
QNetd CA certificate is exported as /etc/corosync/qnetd/nssdb/qnetd-cacert.crt


INFO: copying CA cert and initializing on all nodes
Host key verification failed.


node 'proxmox2': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'proxmox2': Creating new key and cert db
node 'proxmox2': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'proxmox2': Importing CA
INFO: generating cert request
Certificate database doesn't exists. Use /sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n ProxCluster1' failed: exit code 1



Also got this error when I reran everything again with force and deleting all the files and some other stuff, basically I'm just throwing stuff at the keyboard at this point:



Code: 
root@proxR86S:~# pvecm qdevice setup 192.168.0.88 --force
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes
Host key verification failed.

node 'proxmox2': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'proxmox2': Creating new key and cert db
node 'proxmox2': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'proxmox2': Importing CA
INFO: generating cert request
Certificate database doesn't exists. Use /sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n ProxCluster1' failed: exit code 1


While on the other node it looked different. Ran it right after the one above. As you can see this one has error CAHOst key verification failed:

Code: 
root@proxmox2:~# pvecm qdevice setup 192.168.0.88 --force
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed


/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
                (if you think this is a mistake, you may want to use -f option)




INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db


INFO: copying CA cert and initializing on all nodes


node 'proxR86S': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data
node 'proxR86S': Creating new key and cert db
node 'proxR86S': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'proxR86S': Importing CAHost key verification failed.


INFO: generating cert request
Certificate database doesn't exists. Use /sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n ProxCluster1' failed: exit code 1



EDIT: FINALY!!

So what I had to do was also add keys for all hostnames and alternate ways of connecting. Can't figure out which way it was trying to connect so I just went and SSH-ed into all the ways all across everything.

Also changed the hosts file to a completely simple .local everything.
 
Last edited:
  • Like
Reactions: Ramalama
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back