Qdevice / ArchLinux

[nicedevil]

Hey guys, I have a small cluster with just 2 nodes and want to assign a raspberry pi 4 as a qdevice.
This pi is already connected to my mainserver via PiKVM.

Unfurtunately it is running archlinux at default and I struggle on installing the corosync-qnetd package.

I have downloaded the package from the Archpage and unzipped it.
Inside the readme I could find this installation instructions:

Installation
------------
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

On ./configure it is giving me the following:

checking for corosync_common... no
configure: error: Package requirements (libcorosync_common) were not met:

Package 'libcorosync_common', required by 'virtual:world', not found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables corosync_common_CFLAGS
and corosync_common_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.

Unfortunately I can't find a solution to deploy this now

I hope someone can help me out here.

 

[oguz]

hi,

there's AUR package for it [0] which includes the corosync-qnetd service as well

[0]: https://aur.archlinux.org/packages/corosync-qdevice/

 

[nicedevil]

Yeah that was the one I used, but the dependencies were not deployable
I already added armv7h to the PKGBUILD and it is still not working



That is for one of the dependencies called libcgroup (needed by corosync itself)

 

[oguz]

I already added armv7h to the PKGBUILD and it is still not working

but it already has armv7h in the pkgbuild file... EDIT: not in libcgroup i guess -- but you can still build it, make sure you have all the files from the archive

can you try installing it with an AUR client like yay?

try the following:

git clone https://aur.archlinux.org/yay.git
cd yay
makepkg -si # wait for it to complete installing
yay corosync-qdevice

 

[nicedevil]

Ty @oguz , that looked much more promising.

Unfortunately it doesn't seem to be able to start the corosync-qdevice afterwards.

 

[oguz]

Unfortunately it doesn't seem to be able to start the corosync-qdevice afterwards.

the NSS directory is missing, that should be there after the qdevice setup is complete

follow the wiki [0]

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_qdevice_net_setup

 

[nicedevil]

the NSS directory is missing, that should be there after the qdevice setup is complete

follow the wiki [0]

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_qdevice_net_setup

Ty again, but unfortunately it still leads to an error

I installed everything like it is written in the guide on one of my hosts... but it still tells me that the cert couldn't be generated because the certutil seems not to be installed

root@gateway:~# pvecm qdevice setup 10.0.0.51
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
(if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:Sipeg...................................Pq7SX4n5iuggRuEBgQ.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /etc/ssh/ssh_known_hosts:9
remove with:
ssh-keygen -f "/etc/ssh/ssh_known_hosts" -R "10.0.0.22"
RSA host key for 10.0.0.51 has changed and you have requested strict checking.
Host key verification failed.
bash: line 1: corosync-qdevice-net-certutil: command not found

INFO: generating cert request
Certificate database doesn't exists. Use /sbin/corosync-qdevice-net-certutil -i to create it
command 'corosync-qdevice-net-certutil -r -n test1234' failed: exit code 1

Running the missing command seems to be a problem on the archlinux raspberrypi and not on the host after few tests, so I guess the archlinux installation wasn't that successfull as I thought it would be. And it still seems to be installed there :/



EDIT: Ok so as it seems to be there the command, I installed the corosync to my 2nd host as well and run those commands there... a few steps later it looks like this:

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes
Host key verification failed.
command 'ssh -o 'BatchMode=yes' -lroot 10.0.0.51 corosync-qdevice-net-certutil -m -c /etc/pve/qdevice-net-node.p12' failed: exit code 255

 

[nicedevil]

Ok it is getting weird, I think there is an other problem with too much trys to get this up and running.
Right now I'm struggling with the corosync-qdevice-net-certutil -i => that tells me that I should create the certificate database but it is failing with Can't open certificate file

Is there a way to start from completly scratch?

I already tryed with rm -rf /etc/corosync/qnetd/nssdb on the raspberry but still the same error on the certutil

 

[oguz]

INFO: copy and import pk12 cert to all nodes                                                                                                                                                                                                    
Host key verification failed.                                                                                                                                                                                                                  
command 'ssh -o 'BatchMode=yes' -lroot 10.0.0.51 corosync-qdevice-net-certutil -m -c /etc/pve/qdevice-net-node.p12' failed: exit code 255

that part happens because your SSH keys changed. make sure you can ssh to the pi, and that you can ssh to all the other nodes from the pi.

Is there a way to start from completly scratch?

find / -type d 2>/dev/null | grep corosync should give you list of all directories with corosync. on the raspi i'd just wipe all of that and reinstall corosync.

 

[nicedevil]

I did what you told me (as always right now ).... I think I will give up at this point, don't want to waste your time more and more :/

First host is telling me the same:


2nd one tells me this:


I'm able to ssh to the pi from host1 and host2 without the need to enter the password.... so key authentication works as expected.

And the reinstallation after removing all folders like you told me ends with this:



During the installation process I got those warnings:

 

[oguz]

And the reinstallation after removing all folders like you told me ends with this:

i guess not everything was removed then? pacman -Rns <pkgname> could be helpful.

if you don't have anything against using debian for the qdevice then it's definitely going to work (at least that's what we test here)

was the qdevice added in the cluster btw? do you see it in pvecm status output?

from the first two screenshots it looks like an ssh issue though...

if things are still not working i'll take a look at testing it with an archlinux qdevice as well

 

[nicedevil]

i guess not everything was removed then? pacman -Rns <pkgname> could be helpful.

if you don't have anything against using debian for the qdevice then it's definitely going to work (at least that's what we test here)

was the qdevice added in the cluster btw? do you see it in pvecm status output?

from the first two screenshots it looks like an ssh issue though...

if things are still not working i'll take a look at testing it with an archlinux qdevice as well

nope on pvecm status it is still not showing up :/
I guess that is because of the error I got on setting it up.

tryed the removing command as well for archlinux and nope didn't find any package



Because I backed the Kickstarter PiKVM v3 I want to use their provided OS to use everything as it was intended to (except that I want to add qdevice thingi to it ). So sry but debian is no option for that special device :/
I prefer as much as anything is compatible alpine linux and then debian over ubuntu.

 

[WoRie]

nope on pvecm status it is still not showing up :/
I guess that is because of the error I got on setting it up.

tryed the removing command as well for archlinux and nope didn't find any package

View attachment 33437

Because I backed the Kickstarter PiKVM v3 I want to use their provided OS to use everything as it was intended to (except that I want to add qdevice thingi to it ). So sry but debian is no option for that special device :/
I prefer as much as anything is compatible alpine linux and then debian over ubuntu.

Hi,

were you able to get this to work? I also thought, why should my (currently expensive and sold-out) raspberrypi idle around with just pikvm and not also act as quorum to the cluster it's connected to. But using docker didn't work and after reading your post I will refrain from trying to install it bare metal.

 

[baudneo]

Trying to use an arch install as a qdevice and I am getting some of the same errors as the OP:

```
╰─❯ pvecm qdevice setup 10.0.1.9
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed

/bin/ssh-copy-id: WARNING: All keys were skipped because they already exist on the remote system.
(if you think this is a mistake, you may want to use -f option)


INFO: initializing qnetd server
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db

INFO: copying CA cert and initializing on all nodes
zsh:1: command not found: corosync-qdevice-net-certutil
Certificate database already exists. Delete it to continue

INFO: generating cert request
command 'corosync-qdevice-net-certutil -r -n Homelab' failed: open3: exec of corosync-qdevice-net-certutil -r -n Homelab failed: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 455.
```

Any insight?

 

[nicedevil]

Hey guys, sry for not responding here.

I was able in the past to get it up and running but now (maybe it is because PiKVM did something on the OS that I can't figure out) it is not working anymore.

So what I did to get it up and running was basically this what @oguz told me => https://forum.proxmox.com/threads/qdevice-archlinux.102864/post-443008

Sometimes I get a permission denied error on the cache directory. Can be solved with this:

chmod -R 777 /var/cache/private/yay

In the end it looked like this:

[root@pikvm yay]# yay corosync-qdevice
 -> Avoid running yay as root/sudo.
1 aur/corosync-qdevice 3.0.1-1 (+0 0.00) (Out-of-date: 2023-08-07)
    The Corosync Cluster Engine Qdevice
==> Packages to install (eg: 1 2 3, 1-3 or ^4)
==> 1
:: There are 2 providers available for corosync:
:: Repository AUR
    1) corosync 2) corosync-git

Enter a number (default=1): ==> 1
:: Checking for conflicts...
:: Checking for inner conflicts...
[Aur:2]  corosync-3.1.6-1  corosync-qdevice-3.0.1-1

  2 corosync                                 (Build Files Exist)
  1 corosync-qdevice                       
==> Packages to cleanBuild?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==> 1
:: PKGBUILD up to date, Skipping (1/1): corosync
:: (1/1) Downloaded PKGBUILD: corosync-qdevice
  2 corosync                                 (Build Files Exist)
  1 corosync-qdevice                         (Build Files Exist)
==> Diffs to show?
==> [N]one [A]ll [Ab]ort [I]nstalled [No]tInstalled or (1 2 3, 1-3, ^4)
==> 1
diff --git /var/cache/yay/corosync-qdevice/PKGBUILD /var/cache/yay/corosync-qdevice/PKGBUILD
new file mode 100644
index 0000000..206d577
--- /dev/null
+++ /var/cache/yay/corosync-qdevice/PKGBUILD
@@ -0,0 +1,48 @@
+# Maintainer: Patrick Jennings <patrick at jenningsga dot com>
+
+pkgname=corosync-qdevice
+pkgver=3.0.1
+pkgrel=1
+pkgdesc="The Corosync Cluster Engine Qdevice"
+arch=('i686' 'x86_64' 'armv6h' 'armv7h')
+url="http://www.corosync.org/"
+license=('BSD')
+depends=('corosync')
+source=("$pkgname-$pkgver.tar.gz::https://github.com/corosync/$pkgname/archive/v$pkgver.tar.gz")
+sha256sums=('0a8eccd4082fe9deea4819b22515bd1571ed7e36d73b76f3b7570995f2de6834')
+
+prepare() {
+  cd ${pkgname}-${pkgver}
+  ./autogen.sh
+}
+
+build() {
+  cd ${pkgname}-${pkgver}
+  ./configure --sbindir=/usr/bin \
+              --sysconfdir=/etc \
+              --libdir=/usr/lib \
+              --enable-dbus \
+              --enable-monitoring \
+              --enable-watchdog \
+              --enable-systemd \
+              --disable-upstart \
+              --enable-snmp \
+              --enable-xmlconf \
+              --enable-qdevices \
+              --enable-qnetd \
+              --localstatedir=/var \
+              --with-systemddir=/usr/lib/systemd/system \
+              --with-tmpfilesdir=/usr/lib/tmpfiles.d
+  make
+}
+
+package() {
+  cd ${pkgname}-${pkgver}
+
+  make DESTDIR="${pkgdir}" install
+
+  rm -r "$pkgdir/var/run"
+  install -Dm644 LICENSE "$pkgdir/usr/share/licenses/$pkgname/LICENSE"
+}
+
+# vim: set sw=2 et:

==> Proceed with install? [Y/n] 
 -> error merging corosync: fatal: No remote for the current branch.

Hope someone can help with the last line of code here?

EDIT: Ok I think I fixed it again. I did remove everything from the yay folders that belongs to corosync (not needed if you do it first time) and then run through the install process again with `yay corosync-qdevice` if you get the permission errors, use the chmod command above.

After it was installed I did the setup on one of my PVE nodes (`pvecm qdevice setup IPADDRESS`).

Make sure that port 22 and 5403 are opened from PVE to QDevice.

 

[nicedevil]

Trying to use an arch install as a qdevice and I am getting some of the same errors as the OP:

```
INFO: generating cert request
command 'corosync-qdevice-net-certutil -r -n Homelab' failed: open3: exec of corosync-qdevice-net-certutil -r -n Homelab failed: No such file or directory at /usr/share/perl5/PVE/Tools.pm line 455.
```

Any insight?

did you install corosync on all of your nodes before starting?

apt install corosync-qdevice

see here https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_qdevice_net_setup

 

[esi_y]

It's a long thread to follow with references, but a single glance tells me something is very wrong - you need qnetd on the Arch, NOT qdevice, that's service running on the nodes. You are probably doing it the other way around.