JOIN CLUSTER ERROR

[onisokien]

We currently have two datacenters connected to each other through an IPSec Tunnel. We run Proxmox on our servers and we intend to connect them to a cluster. Every time we try joining them into a cluster, we get these errors in the image attached.

 

[fabian]

please provide pveversion -v, network & corosync config as well as the journal output covering the period of the attempted join + some minutes before and after.

 

[onisokien]

Hi, thanks for the reply @fabian.

I am trying on two different servers. I will focus on one and hopefully it works for both.

I have attached the journalctl -f logs of the machine and the machine already existing cluster. I noticed after a while I am unable to login to the web GUI with my password.

Post clustering attempt

 

[fabian]

that doesn't help - I really need the logs during the join from both nodes!

 

[onisokien]

that doesn't help - I really need the logs during the join from both nodes!

Node I am trying to add to cluster (During attempted join)

Nov 21 11:20:00 polymath-prod-pa pvescheduler [117351]: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
Nov 21 11:20:00 polymath-prod-pa pvescheduler [117350]: replication: cfs-lock "file-replication_cfg' error: no quorum!
Nov 21 11:20:01 polymath-prod-pa cron [17171: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Nov 21 11:20:02 polymath-prod-pa corosync [1161771 [TOTEM ] Token has not been received in 6632 ms
Nov 21 11:20:08 polymath-prod-pa corosync [116177] [QUORUM] Sync members [1]: 2
Nov 21 11:20:08 polymath-prod-pa corosync [116177] [TOTEM ] A new membership (2.5ecb1) was formed. Members
Nov 21 11:20:08 polymath-prod-pa corosync [1161771 [QUORUM] Members [1]: 2
Nov 21 11:20:08 polymath-prod-pa corosync [116177] [MAIN] Completed service synchronization, ready to provide service.
Nov 21 11:20:11 polymath-prod-pa corosync [116177] [TOTEM ] Token has not been received in 2912 ms
Nov 21 11:20:14 polymath-prod-pa corosync [116177] [TOTEM 1 Token has not been received in 6641 ms
Nov 21 11:20:20 polymath-prod-pa corosync [116177] [QUORUM] Sync members [1]: 2
Nov 21 11:20:20 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5ecc1) was formed. Members
Nov 21 11:20:20 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:20:20 polymath-prod-pa corosync [116177][MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:20:23 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 2899 ms
Nov 21 11:20:26 polymath-prod-pa corosync [1161771[TOTEM ] Token has not been received in 6627 m
Nov 21 11:20:32 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:20:32 polymath-prod-pa corosync [116177][TOTEM ] A ew membership (2.5eccd) was formed. Members
Nov 21 11:20:32 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:20:32 polymath-prod-pa corosync [1161771[MAIN 1 Completed service synchronization, ready to provide service.
Nov 21 11:20:35 polymath-prod-pa corosync [116177][TOTEM ] Token has not been received in 2991 ms
Nov 21 11:20:38 polymath-prod-pa corosync [116177][TOTEM ] Token has not been received in 6635 ms
Nov 21 11:20:44 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:20:44 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5ecdd) was formed. Members
Nov 21 11:20:44 polymath-prod-pa corosync [1161771[QUORUM] Members [1]: 2
Nov 21 11:20:44 polymath-prod-pa corosync [116177][MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:20:46 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 2998 ms
Nov 21 11:20:50 polymath-prod-pa corosync [116177][TOTEM ] Token has not been received in 6636 ms
Nov 21 11:20:56 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:20:56 polymath-prod-pa corosync [1161771[TOTEM ] A new membership (2.5eced) was formed. Members
Nov 21 11:20:56 polymath-prod-pa corosync [1161771[QUORUM] Members [1]: 2
Nov 21 11:20:56 polymath-prod-pa corosync [1161771[MAIN] Completed service synchronization, ready to provide service.
Nov 21 11:20:58 polymath-prod-pa corosync [116177]:[TOTEM ] Token has not been received in 2906 ms
Nov 21 11:21:00 polymath-prod-pa pvescheduler [117473]: jobs: cfs-lock 'file-jobs cfg' error: no quorum!
Nov 21 11:21:00 polymath-prod-pa pvescheduler [117472]: replication: cfs-lock 'file-replication cfg' error: no quorum!
Nov 21 11:21:01 polymath-prod-pa cron [1717]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Nov 21 11:21:02 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 6634 ms
Nov 21 11:21:07 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:21:07 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5ecfd) was formed. Members
Nov 21 11:21:07 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:21:07 polymath-prod-pa corosync [116177][MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:21:10 polymath-prod-pa corosync [116177]:[TOTEM ] Token has not been received in 2996 ms
Nov 21 11:21:14 polymath-prod-pa corosync [116177][TOTEM ] Token has not been received in 6639 ms
Nov 21 11:21:19 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:21:19 polymath-prod-pa corosync [116177][TOTEM ] A ew membership (2.5eded) was formed. Members
Nov 21 11:21:19 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:21:19 polymath-prod-pa corosync [116177][MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:21:22 polymath-prod-pa corosync [116177][TOTEM ] Token has not been received in 2992 ms
Nov 21 11:21:26 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 6631 ms
Nov 21 11:21:31 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:21:31 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5ed1d) was formed. Members
Nov 21 11:21:31 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:21:31 polymath-prod-pa corosync[1161771MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:21:34 polymath-prod-pa corosync[1161771[TOTEM 1 Token has not been received in 2992 ms
Nov 21 11:21:38 polymath-prod-pa corosync [116177][TOTEM ] Token has not been received in 6630 ms
Nov 21 11:21:43 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:21:43 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5ed2d) was formed. Members
Nov 21 11:21:43 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:21:43 polymath-prod-pa corosync [116177] :[MAIN] Completed service synchronization, ready

Node in the cluster (During attempted join)

Nov 21 11:22:38 polymath login[1384665]: pam unix(login:session): session opened for user root (uid=0) by root (uid=0)
Nov 21 11:22:38 polymath login[1384670]: ROOT LOGIN on"/dev/pts/5' from '192.168.80.253'
Nov 21 11:22:39 polymath corosync[1104]:[QUORUM] Sync members [2]: 1 3
Nov 21 11:22:39 polymath corosync[1104]:[TOTEM ] A new membership (1.5ed79) was formed. Members
Nov 21 11:22:39 polymath corosync 11041:[QUORUM! Sync members [21: 1 3
Nov 21 11:22:39 polymath corosync[1104]:[TOTEM ] A new membership (1.5ed7d) was formed. Members
Nov 21 11:22:39 polymath corosync[1104]:[QUORUM] Members [2]: 1 3
Nov 21 11:22:39 polymath corosync[1104]:[MAIN Completed service synchronization, ready to provide service.
Nov 21 11:22:39 polymath pmxcfs[10931: [statusl notice: received log Nov 21 11:22:39 polymath pmxcfs [1093]: [status] notice: received log
Nov 21 11:22:51 polymath corosync[1104]:[QUORUM] Sync members [2]: 1 3
Nov 21 11:22:51 polymath corosync [1104]:[TOTEM ] A new membership (1.5ed89) was formed. Members
Nov 21 11:22:51 polymath corosync[1104]:[QUORUM] Members [2]: 1 3
Nov 21 11:22:51 polymath corosync[1104]:[MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:22:51 polymath corosync 11041:[QUORUM] Sync members [2]: 1 3
Nov 21 11:22:51 polymath corosync[1104]:[TOTEM ] A new membership (1.5ed8d) was formed. Members
Nov 21 11:22:51 polymath corosync[1104]:[QUORUM] Members [2]: 1 3
Nov 21 11:22:51 polymath corosync[1104]:[MAIN Completed service synchronization, ready to provide service.
Nov 21 11:23:03 polymath corosync[1104]:[QUORUM1 Sync members [21: 1 3
Nov 21 11:23:03 polymath corosync[1104]:[TOTEM ] A new membership (1.5ed99) was formed. Members
Nov 21 11:23:03 polymath corosync [1104]:[QUORUM] Sync members [2]: 1 3
Nov 21 11:23:03 polymath corosync[1104]:[TOTEM I A new membership (1.5ed9d) was formed. Members
Nov 21 11:23:03 polymath corosync[1104]:[QUORUM] Members[21: 1 3
Nov 21 11:23:03 polymath corosync[11041:[MAIN Completed service synchronization, ready to provide service.
Nov 21 11:23:15 polymath corosync 11041:[QUORUM] Sync members [2]: 1 3
Nov 21 11:23:15 polymath corosync[1104]:[TOTEM ] A new membership (1.5eda9) was formed. Members
Nov 21 11:23:15 polymath corosync[1104]:[QUORUM] Sync members [21: 1 3
Nov 21 11:23:15 polymath corosync[1104]:[TOTEM ] A new membership (1.5edad) was formed. Members
Nov 21 11:23:15 polymath corosync[1104]:[QUORUM1 Members [21: 1 3
Nov 21 11:23:15 polymath corosync[1104]:[MAIN 1 Completed service synchronization, ready to provide service.
Nov 21 11:23:26 polymath corosync[1104]:[QUORUM] Sync members [2]: 1 3
Nov 21 11:23:26 polymath corosync[1104]:[TOTEM I A new membership (1.5edb9) was formed. Members
Nov 21 11:23:26 polymath corosync[1104]:[QUORUM] Sync members [21: 1 3
Nov 21 11:23:26 polymath corosync[1104]:[TOTEM ] A new membership (1.5edbd) was formed. Members
Nov 21 11:23:27 polymath corosync 11041:[QUORUM1 Members [21: 1 3
Nov 21 11:23:27 polymath corosync[1104]:[MAIN] Completed service synchronization, ready to provide service.

Node I am trying to add to cluster (When I try accessing GUI after a while of cluster join attempt)

Nov 21 11:37:51 polymath-prod-pa corosync [116177][MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:37:54 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 2995 ms
Nov 21 11:37:58 polymath-prod-pa corosync [116177]:[TOTEM 1 Token has not been received in 6634 ms
Nov 21 11:38:00 polymath-prod-pa pvescheduler [1195371: jobs: cfs-lock 'file-jobs cfg' error: no quorum!
Nov 21 11:38:00 polymath-prod-pa pvescheduler [119536]: replication: cfs-lock 'file-replication cfg' error: no quorum!
Nov 21 11:38:01 polymath-prod-pa cron [1717]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Nov 21 11:38:03 polymath-prod-pa corosync [116177]:[QUORUM] Sync members [1]: 2
Nov 21 11:38:03 polymath-prod-pa corosync [1161771:[TOTEM ] A new membership (2.5f22d) was formed. Members
Nov 21 11:38:03 polymath-prod-pa corosync [1161771[QUORUM] Members [1]: 2
Nov 21 11:38:03 polymath-prod-pa corosync [1161771[MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:38:06 polymath-prod-pa IPCC.x[105769]: pam_unix(proxmox-ve-auth: auth): authentication failure; logname= uid-® euid=® tty= ruser= rhost= user=root
Nov 21 11:38:06 polymath-prod-pa corosync [1161771 :[TOTEM 1 Token has not been received in 2906 ms
Nov 21 11:38:08 polymath-prod-pa pvedaemon [105769]: authentication failure; rhost=::ffff:192.168.90.26 user=root@pam msg Authentication fai lure
Nov 21 11:38:10 polymath-prod-pa corosync [116177]:[TOTEM 1 Token has not been received in 6638 ms
Nov 21 11:38:15 polymath-prod-pa corosync [116177]:[QUORUM] Sync members [1]: 2
Nov 21 11:38:15 polymath-prod-pa corosync [1161771[TOTEM ] A new membership (2.5f23d) was formed. Members
Nov 21 11:38:15 polymath-prod-pa corosync [1161771[QUORUM] Members [1]: 2
Nov 21 11:38:15 polymath-prod-pa corosync [1161771[MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:38:18 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 2903 ms
Nov 21 11:38:22 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 6633 ms
Nov 21 11:38:27 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:38:27 polymath-prod-pa corosync [116177]:[TOTEM ] A new membership (2.5f249) was formed. Members
Nov 21 11:38:27 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:38:27 polymath-prod-pa corosync [1161771[MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:38:30 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 2903 ms
Nov 21 11:38:34 polymath-prod-pa corosync [116177]:[TOTEM 1 Token has not been received in 6632 ms
Nov 21 11:38:38 polymath-prod-pa pvedaemon [105769]: authentication failure; rhost=::ffff: 192.168.90.26 user=root@pam msg=cfs-lock 'authkey' error: no quorum!
Nov 21 11:38:39 polymath-prod-pa corosync [1161771:[QUORUM] Sync members [11: 2
Nov 21 11:38:39 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5259) was formed. Members
Nov 21 11:38:39 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:38:39 polymath-prod-pa corosync [1161771[MAIN] Completed service synchronization, ready to provide service.
Nov 21 11:38:42 polymath-prod-pa corosync [1161771[TOTEM 1 Token has not been received in 2901 ms
Nov 21 11:38:46 polymath-prod-pa corosync [116177][TOTEM 1 Token has not been received in 6632 ms
Nov 21 11:38:51 polymath-prod-pa corosync [116177][QUORUM] Sync members [1]: 2
Nov 21 11:38:51 polymath-prod-pa corosync [116177][TOTEM ] A new membership (2.5£269) was formed. Members
Nov 21 11:38:51 polymath-prod-pa corosync [116177]:[QUORUM] Members [1]: 2
Nov 21 11:38:51 polymath-prod-pa corosync [116177]:[MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:38:54 polymath-prod-pa corosync [1161771[TOTEM 1 Token has not been received in 2903 ms
Nov 21 11:38:58 polymath-prod-pa corosync [1161771:[TOTEM ] Token has not been received in 6636 ms
Nov 21 11:39:00 polymath-prod-pa pvescheduler [1196591: jobs: cfs-lock 'file-jobs_cfg' error: no quorum!
Nov 21 11:39:00 polymath-prod-pa pvescheduler [119658]: replication: cfs-lock 'file-replication cfg' error: no quorum!
Nov 21 11:39:01 polymath-prod-pa cron [1717]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Nov 21 11:39:03 polymath-prod-pa corosync [1161771[QUORUM] Sync members [1]: 2
Nov 21 11:39:03 polymath-prod-pa corosync [116177]:[TOTEM ] A new membership (2.5f279) was formed. Members
Nov 21 11:39:03 polymath-prod-pa corosync [116177][QUORUM] Members [1]: 2
Nov 21 11:39:03 polymath-prod-pa corosync [116177][MAIN ] Completed service synchronization, ready to provide service.
Nov 21 11:39:06 polymath-prod-pa corosync [1161771[TOTEM 1 Token has not been received in 2995 ms

Node I am trying to add to cluster (Post join attempt)

- - Journal begins at Sat 2022-09-03 07:58:00 CST.
Nov 21 11:54:06 polymath-prod-pa pvestatd[1738]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Nov 21 11:54:06 polymath-prod-pa pvestatd[1738]: status update time (9.025 seconds)
Nov 21 11:54:16 polymath-prod-pa pvestatd[1738]: authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Nov 21 11:54:16 polymath-prod-pa pvestatd[17381: status update time (9.025 seconds)
Nov 21 11:54:26 polymath-prod-pa pvestatd[1738]:authkey rotation error: cfs-lock 'authkey' error: got lock request timeout
Nov 21 11:54:26 polymath-prod-pa pvestatd[1738]: status update time (9.026 seconds)
Nov 21 11:54:35 polymath-prod-pa pvestatd[1738]: auth key pair missing, generating new one..
Nov 21 11:54:35 polymath-prod-pa pvestatd[17381: status update time (8.111 seconds)
Nov 21 11:55:01 polymath-prod-pa cron [1717]: (*system*vzdump) CAN'T OPEN SYMLINK (/etc/cron.d/vzdump)
Nov 21 11:55:14 polymath-prod-pa pvecm[121670]: got inotify poll request in wrong process - disabling inotify
Nov 21 11:55:49 polymath-prod-pa pvedaemon [1057691: <root@pam> successful auth for user 'root@pam Nov 21 11:56:01 polymath-prod-pa cron[1717]: (*system*vzdump) RELOAD (/etc/cron.d/vzdump)

Please let me know what else I am missing

 

[fabian]

those logs start while the problem was already ongoing - I really need the logs covering the whole period of the join + some minutes before and after, like I wrote.