[TUTORIAL] FabU: How to setup a Quorum Device and secure it by disabling root access.

UdoB

UdoB

Distinguished Member
Proxmox Subscriber
Nov 1, 2016
3,505
2,213
273
Germany
Statement: it is not required to allow root@cluster to login to a separate “Quorum Device”-machine after installation.

The “tl;dr” boils down to a single line:
Code: 
~$ echo "PermitRootLogin no" | sudo tee /etc/ssh/sshd_config.d/permitrootlogin.conf

(( This is Debian, and it comes without “sudo” by default. I am still trying to convince myself that “passwordless sudo” actually is a great idea... ))

This post is definitely much longer than required. (Perhaps I have too much spare time...) I prefer to leave it in this state as I believe too few details are worse than too many :-)

Quorum - before this experiment​

On a cluster with six nodes I get this:

Code: 
root@pna:~# pvecm status
Cluster information
-------------------
Name:             pn
Config Version:   8
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Mar 10 08:45:42 2026
Quorum provider:  corosync_votequorum
Nodes:            6
Node ID:          0x00000001
Ring ID:          1.16d5
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   6
Highest expected: 6
Total votes:      6
Quorum:           4
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.2.192.1 (local)
0x00000002          1 10.2.192.2
0x00000003          1 10.2.192.3
0x00000004          1 10.2.192.4
0x00000005          1 10.2.192.5
0x00000006          1 10.2.192.6
Code: 
root@pna:~# corosync-cfgtool -n
Local node ID 1, transport knet
nodeid: 2 reachable
   LINK: 0 udp (10.2.192.1->10.2.192.2) enabled connected mtu: 1397
   LINK: 1 udp (192.0.2.1->192.0.2.2) enabled connected mtu: 1397

nodeid: 3 reachable
   LINK: 0 udp (10.2.192.1->10.2.192.3) enabled connected mtu: 1397
   LINK: 1 udp (192.0.2.1->192.0.2.3) enabled connected mtu: 1397

nodeid: 4 reachable
   LINK: 0 udp (10.2.192.1->10.2.192.4) enabled connected mtu: 1397
   LINK: 1 udp (192.0.2.1->192.0.2.4) enabled connected mtu: 1397

nodeid: 5 reachable
   LINK: 0 udp (10.2.192.1->10.2.192.5) enabled connected mtu: 1397
   LINK: 1 udp (192.0.2.1->192.0.2.5) enabled connected mtu: 1397

nodeid: 6 reachable
   LINK: 0 udp (10.2.192.1->10.2.192.6) enabled connected mtu: 1397
   LINK: 1 udp (192.0.2.1->192.0.2.6) enabled connected mtu: 1397


Installation of the OS for a Quorum Device​

This is a pure test-cluster. For productive usage you want to set this up in an off-cluster environment!
  • Debian Trixie via `debian-13.3.0-amd64-netinst.iso`
  • best practice: Debian with a local user “lu” and “root” _without_ a password = only ssh-key login will be possible by default
  • only “SSH Server” gets installed during initial installation
While my test-cluster has access to two networks this VM will have only one NIC. The chosen network has access to the internet, to allow installing software ;-)

The random IP address used here is:
Code: 
lu@trixie-qdev:~$ ip -4 addr show dev ens18
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    altname enp0s18
    altname enxbc2411ffb0ac
    inet 10.2.178.156/16 brd 10.2.255.255 scope global dynamic noprefixroute ens18
       valid_lft 863607sec preferred_lft 755607sec

Note that this is mentioned for documentation purposes only. The QDev is _not_ required to live in the same network as the cluster.

Installation of a Quorum Device​

I am just following https://pve.proxmox.com/pve-docs/chapter-pvecm.html#_corosync_external_vote_support

This VM:
Code: 
lu@trixie-qdev:~$ sudo apt install corosync-qnetd

Cluster:
Code: 
root@pna:~# for H in 1 2 3 4 5 6 ; do echo "---------- $H"; ssh root@10.2.192.$H  "   apt install corosync-qdevice  "; done

Code: 
root@pna:~# pvecm qdevice setup 10.2.178.156
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '10.2.178.156 (10.2.178.156)' can't be established.
ED25519 key fingerprint is SHA256:NgTDm5vFnREahoDEoJjTwOdAhrE4vrV59D+Iyk1QUdg.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.2.178.156's password:

Only now I realized that I need to have "password-auth" for root. Jump to the VM, set one and allow its usage:

This VM:
Code: 
lu@trixie-qdev:~$ sudo passwd
New password:
Retype new password:
passwd: password updated successfully

lu@trixie-qdev:~$ echo "PermitRootLogin yes" | sudo tee /etc/ssh/sshd_config.d/permitrootlogin.conf
PermitRootLogin yes
lu@trixie-qdev:~$ sudo systemctl  restart sshd

Back to the cluster for the second try. (Sorry for the long output - I do not want to crop it here)
Code: 
root@pna:~# pvecm qdevice setup 10.2.178.156
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@10.2.178.156's password:   
                                                                                                                                      
Number of key(s) added: 1
                                                                  
Now try logging into the machine, with: "ssh -i /root/.ssh/id_rsa 'root@10.2.178.156'"
and check to make sure that only the key(s) you wanted were added.
                                                                  
                                                                  
INFO: initializing qnetd server             
Certificate database (/etc/corosync/qnetd/nssdb) already exists. Delete it to initialize new db
                                                                  
INFO: copying CA cert and initializing on all nodes
                                                                  
node 'pna': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data              
node 'pna': Creating new key and cert db       
node 'pna': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pna': Importing CA                
node 'pnb': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data                               
node 'pnb': Creating new key and cert db                                                                                              
node 'pnb': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pnb': Importing CA                                                                                                                                                                                                                                                       
node 'pnc': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data                               
node 'pnc': Creating new key and cert db                                                                                              
node 'pnc': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pnc': Importing CA                                                                                                                                                                                                                                                       
node 'pnd': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data                               
node 'pnd': Creating new key and cert db                                                                                              
node 'pnd': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pnd': Importing CA                                                                                                                                                                                                                                                       
node 'pne': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data                               
node 'pne': Creating new key and cert db                                                                                              
node 'pne': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pne': Importing CA                                                                                                                                                                                                                                                       
node 'pnf': Creating /etc/corosync/qdevice/net/nssdb
password file contains no data                               
node 'pnf': Creating new key and cert db                                                                                              
node 'pnf': Creating new noise file /etc/corosync/qdevice/net/nssdb/noise.txt
node 'pnf': Importing CA                                                                                                                                                                                                                                                       
INFO: generating cert request
Creating new certificate request                             
                                                                                                                                                                                                                                                                               
Generating key.  This may take a few moments...                                                                                                                                                                                                                                
Certificate request stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.crq
                                                                       
INFO: copying exported cert request to qnetd server

INFO: sign and export cluster cert
Signing cluster certificate
Certificate stored in /etc/corosync/qnetd/nssdb/cluster-pn.crt

INFO: copy exported CRT

INFO: import certificate
Importing signed cluster certificate
Notice: Trust flag u is set automatically if the private key is present.
pk12util: PKCS12 EXPORT SUCCESSFUL
Certificate stored in /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12

INFO: copy and import pk12 cert to all nodes

node 'pna': Importing cluster certificate and key
node 'pna': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pnb': Importing cluster certificate and key
node 'pnb': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pnc': Importing cluster certificate and key
node 'pnc': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pnd': Importing cluster certificate and key
node 'pnd': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pne': Importing cluster certificate and key
node 'pne': pk12util: PKCS12 IMPORT SUCCESSFUL
node 'pnf': Importing cluster certificate and key
node 'pnf': pk12util: PKCS12 IMPORT SUCCESSFUL
INFO: add QDevice to cluster configuration

INFO: start and enable corosync qdevice daemon on node 'pna'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.

INFO: start and enable corosync qdevice daemon on node 'pnb'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.

INFO: start and enable corosync qdevice daemon on node 'pnc'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.

INFO: start and enable corosync qdevice daemon on node 'pnd'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.

INFO: start and enable corosync qdevice daemon on node 'pne'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.

INFO: start and enable corosync qdevice daemon on node 'pnf'...
Synchronizing state of corosync-qdevice.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable corosync-qdevice
Created symlink '/etc/systemd/system/multi-user.target.wants/corosync-qdevice.service' -> '/usr/lib/systemd/system/corosync-qdevice.service'.
Reloading corosync.conf...
Done
root@pna:~#

Usage of the Quorum Device​

The limits for long posts are too easily reached --> follow up in post #2





I put this under my “FabU”-label although it was not really “frequently answered” just because... I like it this way ;-)
Please reply if I made something wrong or missed some important details; please “Like” it if you find this post interesting and want to give positive feedback :-)

Thanks for reading!
 
Last edited:
  • Like
Reactions: Johannes S

Usage of the Quorum Device​

It kicks in automatically. Now we got “7” instead of “6” votes = one more than at the beginning:
Code: 
root@pna:~# pvecm status

...

Votequorum information
----------------------
Expected votes:   7
Highest expected: 7
Total votes:      7
Quorum:           4
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 10.2.192.1 (local)
0x00000002          1    A,V,NMW 10.2.192.2
0x00000003          1    A,V,NMW 10.2.192.3
0x00000004          1    A,V,NMW 10.2.192.4
0x00000005          1    A,V,NMW 10.2.192.5
0x00000006          1    A,V,NMW 10.2.192.6
0x00000000          1            Qdevice

Note that Quorum: 4 is not altered.

Security​

This post is about root-access to the QDev and its implications. First let us verify that this is the fact

This VM:
Code: 
lu@trixie-qdev:~$ sudo cat /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCRp3tE3HcISk4Wn5lAQO2LsOuvduR2KJkkJhpmDn4lR/CLNMJjCmi1wGMV/LlGBEoSuHdByKQa7WO6977FnkiAm1Jcp/uDCUhHBf4eB9BCgx1EB81I4z6RI3lCXG+j0Q4liQ0ayo6+dRCxFJGURRcGwZilsJKxQgO7KYETenxA/vIXZf4cIrX65gx2DlxOgZQWqsm0t1chGZBcUGY2ChRSkgPtEVKHkp7aFZjVg7F8Y4aCDBLGSjNS5Pp/SU12WTl3pFPzR+/2K1dqzpq5GZ7mY+L5kmmChsdyICsGDv/KM+TkOekGXOX6b3c3L/sQa08p6Ryv4xcVlrmit77wcYUv root@pna

And thus every Cluster member can run commands like this without knowing the password:
Code: 
root@pna:~# ssh root@10.2.178.156 "hostname; whoami"
trixie-qdev
root

The risk​

Obviously this allows “root” on any cluster node to do everything on the Quorum Device machine. While this is not a drama in itself, it becomes a nightmare if you put the QDev on another system like a PBS - a Proxmox Backup Server.

Thanks @Johannes S for pointing this out: https://forum.proxmox.com/threads/w...-in-a-two-node-pve-cluster.181302/post-841981

In case of a ransomware attack you really want to have your backups untouchable by the attacker!

That post reminded me that this problem popped up already in December 2025: https://forum.proxmox.com/threads/p...in-vm-on-pve-or-not-at-all.177367/post-823814 but the actual behavior was not verified.

The cure​

We do not want “root can access the QDev-VM” because it is simply not necessary anymore. Right? Let’s verify this theory:

The first step is to remove both “password-auth” AND “ssh-key login” for root:

This VM:
Code: 
lu@trixie-qdev:~$ echo "PermitRootLogin no" | sudo tee /etc/ssh/sshd_config.d/permitrootlogin.conf
PermitRootLogin no
lu@trixie-qdev:~$ sudo systemctl  restart sshd

Then try to login:
Code: 
root@pna:~# ssh root@10.2.178.156 "hostname; whoami"
root@10.2.178.156's password:
Permission denied, please try again.

Both key and password does not work anymore. But does the QDev still work? Shut it down and check...

Cluster during shut down QDev:
Code: 
root@pna:~# pvecm status

Votequorum information
----------------------
Expected votes:   7
Highest expected: 7
Total votes:      6
Quorum:           4
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1   A,NV,NMW 10.2.192.1 (local)
0x00000002          1   A,NV,NMW 10.2.192.2
0x00000003          1   A,NV,NMW 10.2.192.3
0x00000004          1   A,NV,NMW 10.2.192.4
0x00000005          1   A,NV,NMW 10.2.192.5
0x00000006          1   A,NV,NMW 10.2.192.6
0x00000000          0            Qdevice (votes 1)

After restart:
Code: 
root@pna:~# pvecm status

Votequorum information
----------------------
Expected votes:   7
Highest expected: 7
Total votes:      7
Quorum:           4
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 10.2.192.1 (local)
0x00000002          1    A,V,NMW 10.2.192.2
0x00000003          1    A,V,NMW 10.2.192.3
0x00000004          1    A,V,NMW 10.2.192.4
0x00000005          1    A,V,NMW 10.2.192.5
0x00000006          1    A,V,NMW 10.2.192.6
0x00000000          1            Qdevice

It works as documented :-)

Future behavior​

Does the removal of “PermitRootLogin” induce any trouble in the future? As long as the relationship on the corosync-level stays established I see no reason. If you see a problem please reply!

Unrelated​

Just because we are experimenting here...

When one or more node fail...?​

What happens? Then we have an even number of votes with the QDev? Let’s try: shutting down node `...192.5` gives me:

Code: 
root@pna:~# pvecm status

Votequorum information
----------------------
Expected votes:   7
Highest expected: 7
Total votes:      6
Quorum:           4
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 10.2.192.1 (local)
0x00000002          1    A,V,NMW 10.2.192.2
0x00000003          1    A,V,NMW 10.2.192.3
0x00000004          1    A,V,NMW 10.2.192.4
0x00000006          1    A,V,NMW 10.2.192.6
0x00000000          1            Qdevice

Kind of expected, right? :-) Stopping two more nodes (`...192.1.` + `...192.3`) requires me to change the console, but it does work exactly as it should:
Code: 
root@pnb:~# pvecm status

Votequorum information
----------------------
Expected votes:   7
Highest expected: 7
Total votes:      4
Quorum:           4
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000002          1    A,V,NMW 10.2.192.2 (local)
0x00000004          1    A,V,NMW 10.2.192.4
0x00000006          1    A,V,NMW 10.2.192.6
0x00000000          1            Qdevice

Now I am down from six nodes to exactly half of it. Without an additional Quorum Device I would have lost said Quorum. But thanks to it the cluster tolerates this and works flawlessly - including automatic “HA”-handling.



I put this under my “FabU”-label although it was not really “frequently answered” just because... I like it this way ;-)
Please reply if I made something wrong or missed some important details; please “Like” it if you find this post interesting and want to give positive feedback :-)

Thanks for reading!
 
Last edited:
  • Like
Reactions: Johannes S
While it's surely a good idea to disable ssh login for root I still think that for even better isolation of the qdevice a setup like the one described by @aaron here is to be oreffered:
aaron

Post in thread '2 Node HA Cluster'

What I maintain personally, are 2 powerful AMD EPYC nodes that run the infrastructure. The 3rd server is far less powerful in terms of CPU and memory and is using cheaper but quite a bit larger SSDs to be used as PBS for that 2-node cluster and also to remote sync another small PVE / PBS combo on the other side of the country.

On that 3rd machine I installed PVE first and PBS on top. PBS is the main functionality for that machine. On the PVE side, for now, the only guest is the qdevice container. That PBS server is there anyway, and using PVE on that one as well helps to keep using the...
  • Like

Basically install the PBS on a PVE single-node with no guests except a qdevice lxc/vm.
 
  • Like
Reactions: UdoB
Johannes S said:
Basically install the PBS on a PVE single-node with no guests except a qdevice lxc/vm.
Click to expand...
Sure. My post says nothing against that approach - disabling ssh for root (originating from the PVE cluster) is okay for a PBS+QDev too :-)
 
  • Like
Reactions: Johannes S
As I said: I agree that it's a good idea to disable root access via ssh to the qdevice. I think it's even better to have a dedicated environment for it, however you set it up ;)
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back