Host not connected to cluster?

qwaszx

New Member
Jan 29, 2009
3
0
1
Hi, I have an issue with one of the machines in my cluster: when I do pveca -l from the cluster master it says "14 : 10.0.0.51 N ERROR: 500 Can't connect to localhost:50011 (connect: Connection refused)". When I use any other node (10.0.0.2) it shows a successful connection. When I pveca -l from the node in question (10.0.0.51) it shows no errors. The web ui says "unable to load cluster table" beside the node. Is there a way I can fix this without taking the cluster down?

From the cluster master:
Code:
h1:~# pveca -l                                                   
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---ROOT---DATA
 1 : 10.0.0.1        M     S  103 days 21:38   0.63    19%     4%     1%
 2 : 10.0.0.12       N     A  103 days 21:49   0.00    22%     3%     1%
 3 : 10.0.0.11       N     A  103 days 21:49   0.01    23%     3%     1%
 4 : 10.0.0.10       N     A  103 days 21:48   0.00    23%     3%     1%
 5 : 10.0.0.9        N     A  103 days 21:48   0.00    22%     3%     1%
 6 : 10.0.0.8        N     A  103 days 21:48   0.00    22%     3%     1%
 7 : 10.0.0.7        N     A  103 days 21:48   0.05    21%     3%     1%
 8 : 10.0.0.6        N     A  103 days 21:48   0.00    21%     3%     1%
 9 : 10.0.0.5        N     A  103 days 21:48   0.01    21%     3%     1%
10 : 10.0.0.4        N     A  103 days 21:48   0.00    22%     3%     1%
11 : 10.0.0.3        N     A  103 days 21:42   0.01    22%     3%     2%
12 : 10.0.0.2        N     A  103 days 21:42   0.00    27%     3%     1%
14 : 10.0.0.51       N     ERROR: 500 Can't connect to localhost:50011 (connect: Connection refused)

15 : 10.0.0.31       N     A   97 days 21:12   0.04    13%     1%     0%
16 : 10.0.0.40       N     A   97 days 20:39   0.01    13%     4%     2%
17 : 10.0.0.39       N     A   97 days 20:08   0.00    13%     4%     2%
18 : 10.0.0.38       N     A   97 days 21:06   0.01    13%     4%     2%
19 : 10.0.0.37       N     A   97 days 21:26   0.02    12%     4%     2%
20 : 10.0.0.34       N     A   97 days 20:25   0.02    13%     4%     2%
21 : 10.0.0.36       N     A   97 days 21:24   0.00    12%     4%     2%
22 : 10.0.0.33       N     A   97 days 20:18   0.04    13%     4%     2%
23 : 10.0.0.32       N     A   97 days 19:57   0.01    13%     4%     2%
From a cluster member:
Code:
h2:~# pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---ROOT---DATA
 1 : 10.0.0.1        M     S  103 days 21:40   0.41    17%     4%     1%
 2 : 10.0.0.12       N     A  103 days 21:50   0.00    22%     3%     1%
 3 : 10.0.0.11       N     A  103 days 21:50   0.00    24%     3%     1%
 4 : 10.0.0.10       N     A  103 days 21:50   0.00    23%     3%     1%
 5 : 10.0.0.9        N     A  103 days 21:50   0.00    22%     3%     1%
 6 : 10.0.0.8        N     A  103 days 21:50   0.00    22%     3%     1%
 7 : 10.0.0.7        N     A  103 days 21:50   0.04    22%     3%     1%
 8 : 10.0.0.6        N     A  103 days 21:50   0.00    21%     3%     1%
 9 : 10.0.0.5        N     A  103 days 21:50   0.00    21%     3%     1%
10 : 10.0.0.4        N     A  103 days 21:50   0.00    22%     3%     1%
11 : 10.0.0.3        N     A  103 days 21:43   0.00    22%     3%     2%
12 : 10.0.0.2        N     A  103 days 21:43   0.00    28%     3%     1%
14 : 10.0.0.51       N     A   46 days 22:48   0.00     3%     1%     0%
15 : 10.0.0.31       N     A   97 days 21:14   0.01    13%     1%     0%
16 : 10.0.0.40       N     A   97 days 20:40   0.00    13%     4%     2%
17 : 10.0.0.39       N     A   97 days 20:10   0.00    13%     4%     2%
18 : 10.0.0.38       N     A   97 days 21:08   0.00    13%     4%     2%
19 : 10.0.0.37       N     A   97 days 21:28   0.10    13%     4%     2%
20 : 10.0.0.34       N     A   97 days 20:26   0.00    13%     4%     2%
21 : 10.0.0.36       N     A   97 days 21:25   0.00    13%     4%     2%
22 : 10.0.0.33       N     A   97 days 20:19   0.01    13%     4%     2%
23 : 10.0.0.32       N     A   97 days 19:59   0.00    13%     4%     2%

From the node which the Master cannot connect to:
Code:
h51:~# pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---ROOT---DATA
 1 : 10.0.0.1        M     S  103 days 21:45   0.42    17%     4%     1%
 2 : 10.0.0.12       N     A  103 days 21:55   0.01    22%     3%     1%
 3 : 10.0.0.11       N     A  103 days 21:55   0.00    23%     3%     1%
 4 : 10.0.0.10       N     A  103 days 21:55   0.00    22%     3%     1%
 5 : 10.0.0.9        N     A  103 days 21:55   0.00    22%     3%     1%
 6 : 10.0.0.8        N     A  103 days 21:55   0.00    22%     3%     1%
 7 : 10.0.0.7        N     A  103 days 21:55   0.01    22%     3%     1%
 8 : 10.0.0.6        N     A  103 days 21:55   0.00    22%     3%     1%
 9 : 10.0.0.5        N     A  103 days 21:55   0.00    22%     3%     1%
10 : 10.0.0.4        N     A  103 days 21:55   0.00    22%     3%     1%
11 : 10.0.0.3        N     A  103 days 21:49   0.00    22%     3%     2%
12 : 10.0.0.2        N     A  103 days 21:49   0.00    28%     3%     1%
14 : 10.0.0.51       N     A   46 days 22:53   0.00     4%     1%     0%
15 : 10.0.0.31       N     A   97 days 21:19   0.00    13%     1%     0%
16 : 10.0.0.40       N     A   97 days 20:45   0.00    13%     4%     2%
17 : 10.0.0.39       N     A   97 days 20:15   0.00    13%     4%     2%
18 : 10.0.0.38       N     A   97 days 21:13   0.06    13%     4%     2%
19 : 10.0.0.37       N     A   97 days 21:33   0.08    13%     4%     2%
20 : 10.0.0.34       N     A   97 days 20:31   0.00    13%     4%     2%
21 : 10.0.0.36       N     A   97 days 21:30   0.01    13%     4%     2%
22 : 10.0.0.33       N     A   97 days 20:24   0.00    13%     4%     2%
23 : 10.0.0.32       N     A   97 days 20:04   0.12    13%     4%     2%
 
check if /etc/pve/cluster.cfg is the same on the master and the out-of-sync node.

then please stop on both the following services.

  • /etc/init.d/pvetunnel stop
  • /etc/init.d/pvemirror stop
  • /etc/init.d/pvedaemon stop
and start again:

  • /etc/init.d/pvetunnel start
  • /etc/init.d/pvemirror start
  • /etc/init.d/pvedaemon start
 
The config files were identical but restarting on the cluster master fixed it. Thanks :-)