Hey all, I just recently reconfigured my qdevice (I'm running a two-node setup with an external qdevice, per https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support).
When I install `corosync-qnetd` on the external qdevice (Raspberry Pi running the latest Raspbian), I never get any votes:
When I look at the corosync-qnetd.service logs on the Pi, I see a CA cert issue:
So i tried the usual `pvecm updatecerts --force` route, just to make sure something didn't get weird with my certificates, and the problem persists (even uninstalled/reinstalled the qdevice).
`corosync-qdevice` on the host itself throws the same errors, presumably as the same error - just server-side:
Code:
root@ff-pve02:~# pveversion -v
proxmox-ve: 7.4-1 (running kernel: 5.15.107-2-pve)
pve-manager: 7.4-4 (running version: 7.4-4/4a8501a8)
pve-kernel-5.15: 7.4-3
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-1
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.6
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
openvswitch-switch: 2.15.0+ds1-2+deb11u4
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.0
pve-cluster: 7.3-3
pve-container: 4.4-4
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-2
pve-firewall: 4.3-2
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1When I install `corosync-qnetd` on the external qdevice (Raspberry Pi running the latest Raspbian), I never get any votes:
Code:
root@ff-pve01:~# pvecm status
Cluster information
-------------------
Name: ff-pve
Config Version: 13
Transport: knet
Secure auth: on
Quorum information
------------------
Date: Tue Jul 18 12:35:17 2023
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000001
Ring ID: 1.62
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate Qdevice
Membership information
----------------------
Nodeid Votes Qdevice Name
0x00000001 1 A,NV,NMW 10.1.0.31 (local)
0x00000002 1 A,NV,NMW 10.1.0.32
0x00000000 0 Qdevice (votes 1)When I look at the corosync-qnetd.service logs on the Pi, I see a CA cert issue:
Code:
ryanb@zigbee:~ $ sudo service corosync-qnetd status
● corosync-qnetd.service - Corosync Qdevice Network daemon
Loaded: loaded (/lib/systemd/system/corosync-qnetd.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-07-18 18:15:21 BST; 15min ago
Docs: man:corosync-qnetd
Main PID: 649 (corosync-qnetd)
Tasks: 1 (limit: 1599)
CPU: 11.920s
CGroup: /system.slice/corosync-qnetd.service
└─649 /usr/bin/corosync-qnetd -f
Jul 18 18:30:12 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:12 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:15 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:15 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:19 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:19 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:21 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:22 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:23 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.
Jul 18 18:30:24 zigbee corosync-qnetd[649]: Unhandled error when reading from client. Disconnecting client (-12195): Peer does not recognize and trust the CA that issued your certificate.So i tried the usual `pvecm updatecerts --force` route, just to make sure something didn't get weird with my certificates, and the problem persists (even uninstalled/reinstalled the qdevice).
`corosync-qdevice` on the host itself throws the same errors, presumably as the same error - just server-side:
Code:
root@ff-pve01:~# service corosync-qdevice status
● corosync-qdevice.service - Corosync Qdevice daemon
Loaded: loaded (/lib/systemd/system/corosync-qdevice.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-07-18 12:30:32 CDT; 7min ago
Docs: man:corosync-qdevice
Main PID: 1565947 (corosync-qdevic)
Tasks: 2 (limit: 154463)
Memory: 2.1M
CPU: 1.086s
CGroup: /system.slice/corosync-qdevice.service
├─1565947 /usr/sbin/corosync-qdevice -f
└─1566099 /usr/sbin/corosync-qdevice -f
Jul 18 12:37:35 ff-pve01 corosync-qdevice[1565947]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:37:35 ff-pve01 corosync-qdevice[1565947]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:37:35 ff-pve01 corosync-qdevice[1565947]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:37:35 ff-pve01 corosync-qdevice[1565947]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:37:37 ff-pve01 corosync-qdevice[1565947]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:37:37 ff-pve01 corosync-qdevice[1565947]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:37:41 ff-pve01 corosync-qdevice[1565947]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:37:41 ff-pve01 corosync-qdevice[1565947]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:37:42 ff-pve01 corosync-qdevice[1565947]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:37:42 ff-pve01 corosync-qdevice[1565947]: Unhandled error when reading from server. Disconnecting from server
#########################################################################################################################################
root@ff-pve02:~# service corosync-qdevice status
● corosync-qdevice.service - Corosync Qdevice daemon
Loaded: loaded (/lib/systemd/system/corosync-qdevice.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2023-07-18 12:30:39 CDT; 5min ago
Docs: man:corosync-qdevice
Main PID: 246775 (corosync-qdevic)
Tasks: 2 (limit: 154275)
Memory: 1.8M
CPU: 736ms
CGroup: /system.slice/corosync-qdevice.service
├─246775 /usr/sbin/corosync-qdevice -f
└─246778 /usr/sbin/corosync-qdevice -f
Jul 18 12:36:07 ff-pve02 corosync-qdevice[246775]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:36:07 ff-pve02 corosync-qdevice[246775]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:36:08 ff-pve02 corosync-qdevice[246775]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:36:08 ff-pve02 corosync-qdevice[246775]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:36:10 ff-pve02 corosync-qdevice[246775]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:36:10 ff-pve02 corosync-qdevice[246775]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:36:13 ff-pve02 corosync-qdevice[246775]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:36:13 ff-pve02 corosync-qdevice[246775]: Unhandled error when reading from server. Disconnecting from server
Jul 18 12:36:15 ff-pve02 corosync-qdevice[246775]: Server certificate verification failure. (-8172): Peer's certificate issuer has been marked as not trusted by the user.
Jul 18 12:36:15 ff-pve02 corosync-qdevice[246775]: Unhandled error when reading from server. Disconnecting from server
Last edited: