Proxmox cluster

R

Raphael Ferreira

New Member
May 19, 2025
11
0
1
I'm having an issue when trying to add a Proxmox node to a cluster. We've identified that the SSH keys are not being exchanged properly, and when we attempt to join the node to the cluster, the process fails and does not complete successfully.
 
Raphael Ferreira said:
I'm having an issue when trying to add a Proxmox node to a cluster. We've identified that the SSH keys are not being exchanged properly, and when we attempt to join the node to the cluster, the process fails and does not complete successfully.
Click to expand...
root@pve6:~# pvecm updatecerts
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
got timeout when trying to ensure cluster certificates and base file hierarchy is set up - no quorum (yet) or hung pmxcfs?

pvecm expected 1

root@pve6:~# pvecm updatecerts
(re)generate node files
Could not read private key from /etc/pve/priv/pve-root-ca.key
generating pve root certificate failed:
command 'faketime yesterday openssl req -batch -days 3650 -new -x509 -nodes -key /etc/pve/priv/pve-root-ca.key -out /etc/pve/pve-root-ca.pem -subj '/CN=Proxmox Virtual Environment/OU=6551d135-81fd-4854-a5ef-e2ed838514be/O=PVE Cluster Manager CA/'' failed: exit code 1
 
bbgeek17 said:
Check your MTU settings and their effectiveness (large packet ICMP) across the board.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
root@pve6:~# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 90:b1:1c:10:49:93 brd ff:ff:ff:ff:ff:ff
altname enp1s0f0
3: eno2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 90:b1:1c:10:49:94 brd ff:ff:ff:ff:ff:ff
altname enp1s0f1
4: enp4s12f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:09:0f:ba:49:3a brd ff:ff:ff:ff:ff:ff
5: enp4s12f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 00:09:0f:ba:49:3b brd ff:ff:ff:ff:ff:ff
6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 90:b1:1c:10:49:93 brd ff:ff:ff:ff:ff:ff
7: eno1.10@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vmbr1 state UP mode DEFAULT group default qlen 1000
link/ether 90:b1:1c:10:49:93 brd ff:ff:ff:ff:ff:ff
8: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 90:b1:1c:10:49:93 brd ff:ff:ff:ff:ff:ff
root@pve6:~#
 
bbgeek17 said:
Check your MTU settings and their effectiveness (large packet ICMP) across the board.

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
root@pve6:~# ping -M do -s 1472 100.64.60.2
PING 100.64.60.2 (100.64.60.2) 1472(1500) bytes of data.
1480 bytes from 100.64.60.2: icmp_seq=1 ttl=60 time=0.406 ms
1480 bytes from 100.64.60.2: icmp_seq=2 ttl=60 time=0.441 ms
1480 bytes from 100.64.60.2: icmp_seq=3 ttl=60 time=0.410 ms
1480 bytes from 100.64.60.2: icmp_seq=4 ttl=60 time=0.448 ms
^C
--- 100.64.60.2 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3052ms
rtt min/avg/max/mdev = 0.406/0.426/0.448/0.018 ms
root@pve6:~# ping -M do -s 1472 100.64.60.6
PING 100.64.60.6 (100.64.60.6) 1472(1500) bytes of data.
1480 bytes from 100.64.60.6: icmp_seq=1 ttl=61 time=0.397 ms
1480 bytes from 100.64.60.6: icmp_seq=2 ttl=61 time=0.457 ms
1480 bytes from 100.64.60.6: icmp_seq=3 ttl=61 time=0.322 ms
1480 bytes from 100.64.60.6: icmp_seq=4 ttl=61 time=0.331 ms
^C
--- 100.64.60.6 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3054ms
rtt min/avg/max/mdev = 0.322/0.376/0.457/0.054 ms
root@pve6:~#
 
Perhaps someone can assist once you provide more information:

- pvecm status
- pvecm nodes (from each node)
- pveversion (across the cluster if it differs)
- command line join with full CLI and output, if the node is half-joined - remove it first and try to start from a clean slate.
- journalctl from each node

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
bbgeek17 said:
Perhaps someone can assist once you provide more information:

- pvecm status
- pvecm nodes (from each node)
- pveversion (across the cluster if it differs)
- command line join with full CLI and output, if the node is half-joined - remove it first and try to start from a clean slate.
- journalctl from each node

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
root@pve4:~# pvecm status
Cluster information
-------------------
Name: Worldnet-DC
Config Version: 9
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Thu May 29 15:35:53 2025
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000001
Ring ID: 1.8f
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 100.64.60.10 (local)
0x00000002 1 100.64.60.9



root@pve5:~# pvecm status
Cluster information
-------------------
Name: Worldnet-DC
Config Version: 9
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Thu May 29 15:36:45 2025
Quorum provider: corosync_votequorum
Nodes: 2
Node ID: 0x00000002
Ring ID: 1.8f
Quorate: Yes

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 2
Quorum: 2
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 100.64.60.10
0x00000002 1 100.64.60.9 (local)



root@pve6:~# pvecm status
Cluster information
-------------------
Name: Worldnet-DC
Config Version: 9
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Thu May 29 15:37:32 2025
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000003
Ring ID: 3.f
Quorate: No

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000003 1 100.64.60.11 (local)




root@pve4:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 pve4 (local)
2 1 pve5


root@pve5:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
1 1 pve4
2 1 pve5 (local)



root@pve6:~# pvecm nodes

Membership information
----------------------
Nodeid Votes Name
3 1 pve6 (local)




root@pve4:~# pveversion
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-11-pve)



root@pve5:~# pveversion
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-11-pve)



root@pve6:~# pveversion
pve-manager/8.4.1/2a5fa54a8503f96d (running kernel: 6.8.12-10-pve)




root@pve4:~# journalctl -xe
May 29 15:41:22 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:23 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:24 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:25 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:26 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:27 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:28 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:29 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:30 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:31 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:32 pve4 pvedaemon[2488695]: <root@pam> end task UPID:pve4:002DDC36:00F84C67:6838A9EB:vncshell::root@pam: OK
May 29 15:41:32 pve4 pvedaemon[3008478]: starting termproxy UPID:pve4:002DE7DE:00F87868:6838AA5C:vncshell::root@pam:
May 29 15:41:32 pve4 pvedaemon[2488695]: <root@pam> starting task UPID:pve4:002DE7DE:00F87868:6838AA5C:vncshell::root@pam:
May 29 15:41:32 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:32 pve4 pvedaemon[2831014]: <root@pam> successful auth for user 'root@pam'
May 29 15:41:32 pve4 login[3008481]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
May 29 15:41:32 pve4 systemd-logind[3634]: New session 12303 of user root.
░░ Subject: A new session 12303 has been created for user root
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ Documentation: sd-login(3)
░░
░░ A new session with the ID 12303 has been created for the user root.
░░
░░ The leading process of the session is 3008481.
May 29 15:41:32 pve4 systemd[1]: Started session-12303.scope - Session 12303 of User root.
░░ Subject: A start job for unit session-12303.scope has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit session-12303.scope has finished successfully.
░░
░░ The job identifier is 1211815.
May 29 15:41:32 pve4 login[3008486]: ROOT LOGIN on '/dev/pts/4'
May 29 15:41:33 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:34 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:35 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:41:36 pve4 corosync[4603]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
lines 2657-2694/2694 (END)




root@pve5:~# journalctl -xe
May 29 15:42:04 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:05 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:06 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:07 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:08 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:09 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:10 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:11 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:12 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:13 pve5 pmxcfs[4101]: [status] notice: received log
May 29 15:42:13 pve5 pmxcfs[4101]: [status] notice: received log
May 29 15:42:13 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:14 pve5 pmxcfs[4101]: [status] notice: received log
May 29 15:42:14 pve5 sshd[3887689]: Accepted publickey for root from 179.189.249.254 port 45722 ssh2: RSA SHA256:0YrkoZN6F4wiMwf9ac9Gzr08BuiDxpa6RtJr8kekaqk
May 29 15:42:14 pve5 sshd[3887689]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
May 29 15:42:14 pve5 systemd-logind[3653]: New session 583 of user root.
░░ Subject: A new session 583 has been created for user root
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ Documentation: sd-login(3)
░░
░░ A new session with the ID 583 has been created for the user root.
░░
░░ The leading process of the session is 3887689.
May 29 15:42:14 pve5 systemd[1]: Started session-583.scope - Session 583 of User root.
░░ Subject: A start job for unit session-583.scope has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit session-583.scope has finished successfully.
░░
░░ The job identifier is 44760.
May 29 15:42:14 pve5 sshd[3887689]: pam_env(sshd:session): deprecated reading of user environment enabled
May 29 15:42:14 pve5 login[3887697]: pam_unix(login:session): session opened for user root(uid=0) by root(uid=0)
May 29 15:42:14 pve5 login[3887702]: ROOT LOGIN on '/dev/pts/1' from '179.189.249.254'
May 29 15:42:14 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:15 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:16 pve5 corosync[4219]: [KNET ] rx: Packet rejected from 187.1.167.225:5405



root@pve6:~# journalctl -xe
May 29 15:42:36 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:36 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:36 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:37 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:37 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:37 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:38 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:38 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:39 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:39 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:40 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:40 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:40 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:41 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:41 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:41 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:42 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:42 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:43 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:43 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:44 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:44 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:44 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:45 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:45 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:45 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:46 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 15:42:46 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:40600
May 29 15:42:47 pve6 corosync[1401]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
lines 1108-1136/1136 (END)
 
bbgeek17 said:
My guess is that the network that the nodes are trying to use to communicate is not what you think it should be. Clearly there is an attempt here to use your internet facing public IPs. Possibly due to an incorrect DNS resolution (including local hosts file).

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
root@pve6:~# journalctl -u pve-cluster -b | tail -n 50
May 29 15:45:11 pve6 pmxcfs[1404]: [confdb] crit: cmap_dispatch failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [quorum] crit: quorum_dispatch failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [dcdb] crit: cpg_dispatch failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [dcdb] crit: cpg_leave failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [status] crit: cpg_dispatch failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [status] crit: cpg_leave failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [quorum] crit: quorum_initialize failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [quorum] crit: can't initialize service
May 29 15:45:11 pve6 pmxcfs[1404]: [confdb] crit: cmap_initialize failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [confdb] crit: can't initialize service
May 29 15:45:11 pve6 pmxcfs[1404]: [dcdb] notice: start cluster connection
May 29 15:45:11 pve6 pmxcfs[1404]: [dcdb] crit: cpg_initialize failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [dcdb] crit: can't initialize service
May 29 15:45:11 pve6 pmxcfs[1404]: [status] notice: start cluster connection
May 29 15:45:11 pve6 pmxcfs[1404]: [status] crit: cpg_initialize failed: 2
May 29 15:45:11 pve6 pmxcfs[1404]: [status] crit: can't initialize service
May 29 15:45:12 pve6 pmxcfs[1404]: [quorum] crit: quorum_finalize failed: 9
May 29 15:45:12 pve6 pmxcfs[1404]: [confdb] crit: cmap_track_delete nodelist failed: 9
May 29 15:45:12 pve6 pmxcfs[1404]: [confdb] crit: cmap_track_delete version failed: 9
May 29 15:45:12 pve6 pmxcfs[1404]: [confdb] crit: cmap_finalize failed: 9
May 29 15:45:12 pve6 pmxcfs[1404]: [main] notice: exit proxmox configuration filesystem (0)
May 29 15:45:12 pve6 systemd[1]: pve-cluster.service: Deactivated successfully.
May 29 15:45:12 pve6 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 29 15:45:54 pve6 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
May 29 15:45:54 pve6 pmxcfs[3096]: [main] notice: resolved node name 'pve6' to '100.64.60.26' for default node IP address
May 29 15:45:54 pve6 pmxcfs[3096]: [main] notice: resolved node name 'pve6' to '100.64.60.26' for default node IP address
May 29 15:45:55 pve6 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
May 29 16:07:39 pve6 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
May 29 16:07:39 pve6 pmxcfs[3097]: [main] notice: teardown filesystem
May 29 16:07:40 pve6 pmxcfs[3097]: [main] notice: exit proxmox configuration filesystem (0)
May 29 16:07:40 pve6 systemd[1]: pve-cluster.service: Deactivated successfully.
May 29 16:07:40 pve6 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
May 29 16:07:40 pve6 systemd[1]: pve-cluster.service: Consumed 1.282s CPU time.
May 29 16:07:40 pve6 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
May 29 16:07:40 pve6 pmxcfs[6372]: [main] notice: resolved node name 'pve6' to '100.64.60.26' for default node IP address
May 29 16:07:40 pve6 pmxcfs[6372]: [main] notice: resolved node name 'pve6' to '100.64.60.26' for default node IP address
May 29 16:07:40 pve6 pmxcfs[6374]: [quorum] crit: quorum_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [quorum] crit: can't initialize service
May 29 16:07:40 pve6 pmxcfs[6374]: [confdb] crit: cmap_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [confdb] crit: can't initialize service
May 29 16:07:40 pve6 pmxcfs[6374]: [dcdb] crit: cpg_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [dcdb] crit: can't initialize service
May 29 16:07:40 pve6 pmxcfs[6374]: [status] crit: cpg_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [status] crit: can't initialize service
May 29 16:07:41 pve6 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
May 29 16:07:46 pve6 pmxcfs[6374]: [status] notice: update cluster info (cluster name Worldnet-DC, version = 11)
May 29 16:07:46 pve6 pmxcfs[6374]: [dcdb] notice: members: 3/6374
May 29 16:07:46 pve6 pmxcfs[6374]: [dcdb] notice: all data is up to date
May 29 16:07:46 pve6 pmxcfs[6374]: [status] notice: members: 3/6374
May 29 16:07:46 pve6 pmxcfs[6374]: [status] notice: all data is up to date



May 29 16:07:40 pve6 pmxcfs[6372]: [main] notice: resolved node name 'pve6' to '100.64.60.26' for default node IP addre>
May 29 16:07:40 pve6 pmxcfs[6372]: [main] notice: resolved node name 'pve6' to '100.64.60.26' for default node IP addre>
May 29 16:07:40 pve6 pmxcfs[6374]: [quorum] crit: quorum_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [quorum] crit: can't initialize service
May 29 16:07:40 pve6 pmxcfs[6374]: [confdb] crit: cmap_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [confdb] crit: can't initialize service
May 29 16:07:40 pve6 pmxcfs[6374]: [dcdb] crit: cpg_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [dcdb] crit: can't initialize service
May 29 16:07:40 pve6 pmxcfs[6374]: [status] crit: cpg_initialize failed: 2
May 29 16:07:40 pve6 pmxcfs[6374]: [status] crit: can't initialize service
May 29 16:07:41 pve6 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
░░ Subject: A start job for unit pve-cluster.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pve-cluster.service has finished successfully.
░░
░░ The job identifier is 1194.
May 29 16:07:46 pve6 pmxcfs[6374]: [status] notice: update cluster info (cluster name Worldnet-DC, version = 11)
May 29 16:07:46 pve6 pmxcfs[6374]: [dcdb] notice: members: 3/6374
May 29 16:07:46 pve6 pmxcfs[6374]: [dcdb] notice: all data is up to date
May 29 16:07:46 pve6 pmxcfs[6374]: [status] notice: members: 3/6374
May 29 16:07:46 pve6 pmxcfs[6374]: [status] notice: all data is up to date



root@pve6:~# journalctl -xeu corosync
May 29 16:11:20 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:21 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:21 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:22 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:22 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:23 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:23 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:24 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:24 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:24 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:24 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:25 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:25 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:26 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:26 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:27 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:27 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:28 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:28 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:28 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:28 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:29 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:29 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:30 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:30 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 16:11:31 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 16:11:31 pve6 corosync[6371]: [KNET ] rx: Packet rejected from 187.1.167.225:5405


How can I solve this issue?
 
bbgeek17 said:
Review your configuration. At the very least:
/etc/pve/corosync.cfg
/etc/hosts
hostnamectl
Check your name resolution: ping, curl, getent hosts [host]

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
Click to expand...
root@pve4:~# cat /etc/pve/corosync.conf
logging {
debug: off
to_syslog: yes
}

nodelist {
node {
name: pve4
nodeid: 1
quorum_votes: 1
ring0_addr: 100.64.60.10
}
node {
name: pve5
nodeid: 2
quorum_votes: 1
ring0_addr: 100.64.60.9
}
node {
name: pve6
nodeid: 3
quorum_votes: 1
ring0_addr: 100.64.60.11
}
}

quorum {
provider: corosync_votequorum
}

totem {
cluster_name: Worldnet-DC
config_version: 11
interface {
linknumber: 0
}
ip_version: ipv4-6
link_mode: passive
secauth: on
version: 2
}



root@pve4:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
100.64.60.6 pve4.worldnet.com.br pve4
100.64.60.2 pve5.worldnet.com.br pve5
100.64.60.26 pve6.worldnet.com.br pve6

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts


root@pve5:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
100.64.60.2 pve5.worldnet.com.br pve5
100.64.60.6 pve4.worldnet.com.br pve4
100.64.60.26 pve6.worldnet.com.br pve6

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts



root@pve6:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
100.64.60.26 pve6.worldnet.com.br pve6
100.64.60.6 pve4.worldnet.com.br pve4
100.64.60.2 pve5.worldnet.com.br pve5

# The following lines are desirable for IPv6 capable hosts

::1 pve6.worldnet.com.br pve6
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts



root@pve6:~# hostnamectl
Static hostname: pve6
Icon name: computer-server
Chassis: server
Machine ID: 2d01d58f5f664bcfb71840023c3c81a2
Boot ID: 62870bd7eaf74ea0bd03d71c8af79d61
Operating System: Debian GNU/Linux 12 (bookworm)
Kernel: Linux 6.8.12-10-pve
Architecture: x86-64
Hardware Vendor: Dell Inc.
Hardware Model: PowerEdge R410
Firmware Version: 1.10.2
 
May be your Colo does erroneous NAT between the hosts and translates the IPs improperly... Only you have access to the entirety of the infrastructure to understand all the intricacies. Network captures may be helpful.

The issue is most likely specific to your infrastructure. Perhaps purchasing a support subscription and engaging dedicated support personnel will be more efficient.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
SteveITS said:
100.64.0.0/10 is CGNAT address space.

How is 187.1.167.225 involved?

You could try running "corosync-cfgtool -n" on each, which shows how the cluster is trying to connect.
Click to expand...
root@pve6:~# journalctl -f
May 29 17:42:41 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 17:42:41 pve6 pveproxy[20807]: worker exit
May 29 17:42:41 pve6 pveproxy[998]: worker 20807 finished
May 29 17:42:41 pve6 pveproxy[998]: starting 1 worker(s)
May 29 17:42:41 pve6 pveproxy[998]: worker 20834 started
May 29 17:42:41 pve6 pveproxy[20834]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2088.
May 29 17:42:42 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 17:42:42 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 17:42:43 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 17:42:43 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 17:42:43 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 17:42:44 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
May 29 17:42:44 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:5405
May 29 17:42:44 pve6 corosync[9256]: [KNET ] rx: Packet rejected from 187.1.167.225:63111
^C
root@pve6:~# cd /etc/pve/
root@pve6:/etc/pve# ls
corosync.conf local lxc openvz qemu-server
root@pve6:/etc/pve# pvecm status
Cluster information
-------------------
Name: Worldnet-DC
Config Version: 11
Transport: knet
Secure auth: on

Quorum information
------------------
Date: Thu May 29 17:44:56 2025
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000003
Ring ID: 3.19
Quorate: No

Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 1
Quorum: 2 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000003 1 100.64.60.11 (local)



root@pve6:/etc/pve# corosync-cfgtool -n
Local node ID 3, transport knet
 
Raphael Ferreira said:
root@pve6:/etc/pve# corosync-cfgtool -n
Local node ID 3, transport knet
Click to expand...
Run that on a working node. Unless it's a paste-o you're missing all the info.

Code: 
# corosync-cfgtool -n
Local node ID 5, transport knet
nodeid: 1 reachable
   LINK: 0 udp (...->...) enabled connected mtu: 1397
   LINK: 1 udp (...->...) enabled connected mtu: 8885
   LINK: 2 udp (...->...) enabled connected mtu: 1397
  
(and then the rest)

Do the corosync IP subnet(s) exist on pve6?
 
SteveITS said:
Run that on a working node. Unless it's a paste-o you're missing all the info.

Code: 
# corosync-cfgtool -n
Local node ID 5, transport knet
nodeid: 1 reachable
   LINK: 0 udp (...->...) enabled connected mtu: 1397
   LINK: 1 udp (...->...) enabled connected mtu: 8885
   LINK: 2 udp (...->...) enabled connected mtu: 1397
 
(and then the rest)

Do the corosync IP subnet(s) exist on pve6?
Click to expand...
root@pve4:/etc/pve/priv# corosync-cfgtool -n
Local node ID 1, transport knet
nodeid: 2 reachable
LINK: 0 udp (100.64.60.10->100.64.60.9) enabled connected mtu: 1397



root@pve5:~# corosync-cfgtool -n
Local node ID 2, transport knet
nodeid: 1 reachable
LINK: 0 udp (100.64.60.9->100.64.60.10) enabled connected mtu: 1397


root@pve6:/etc/pve# corosync-cfgtool -n
Local node ID 3, transport knet

I have already tested ping and telnet and there is communication between the machines pve4 pve5 and pve6
 
I suppose it may just be a symptom, because you started out saying the node didn't join properly.

Best practice is to have more than one corosync link, if possible.

Just to ask is the root password correct? The join process should log in and add its keys, as I understand it.

Adding the node to the cluster overwrites the local web cert with the cluster's default cert.
 
It seems that pve4 and pve5 are not aware of pve6. Follow the guide to clear the cluster configuration/files on pve6, reboot it, then try to rejoin. Carefully record all steps and their output. I recommend doing the join from CLI to ease the data collection.

You also never addressed what this IP is: 187.1.167.225
Where does it come from?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Might be a good idea to compare all your /etc/network/interfaces files on all nodes, as well as the hosts files. Make sure you have at least one dedicated interface for corosync (two preferred) and that all nodes can ping each other on the corosync interface(s.) good practice to also create a hosts file that contains all pve short names resolving to their primary corosync ip address and deploy it to all nodes.
 
You must log in or register to reply here.
Top Bottom