Unable to load local cluster table

now I have tried to change nodes name and ip.

result on master:

CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---ROOT---DATA
1 : 123.0.123.30 M S 22 days 21:01 1.67 58% 19% 86%
2 : 123.0.123.20 N S 262 days 16:21 0.34 61% 6% 16%
3 : 123.0.123.40 N S 22 days 20:31 0.09 81% 5% 84%
7 : 123.0.123.21 N ERROR: 400 Invalid SOAPAction


result on node:

node7:~# pveca -l
CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---DISK
1 : 123.0.123.30 M S 22 days 21:01 1.05 56% 19%
2 : 123.0.123.20 N S 262 days 16:20 0.35 61% 6%
3 : 123.0.123.40 N S 22 days 20:30 0.11 81% 5%
7 : 123.0.123.21 N A 00:12 0.65 12% 4%



:confused:

I become crazy. Please help!
 
Last edited:
maybe this will help. log from master:

Jun 18 13:59:07 syslog0 last message repeated 9 timesJun 18 13:59:08 pvemirror25901 starting cluster syncronization Jun 18 13:59:08 pvemirror25901 syncing vzlist from '123.0.123.21' failed: 400 Invalid SOAPAction Jun 18 13:59:08 pvemirror25901 syncing templates Jun 18 13:59:08 pvemirror25901 cluster syncronization finished (0.52 seconds (files 0.00, config 0.00)) Jun 18 13:59:08 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:13 syslog0 last message repeated 5 timesJun 18 13:59:14 proxwww8135 Starting new child 8135 Jun 18 13:59:14 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:19 syslog0 last message repeated 5 timesJun 18 13:59:20 proxwww8136 Starting new child 8136 Jun 18 13:59:20 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:25 syslog0 last message repeated 5 timesJun 18 13:59:26 proxwww8137 Starting new child 8137 Jun 18 13:59:26 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:40 syslog0 last message repeated 14 timesJun 18 13:59:41 proxwww8151 Starting new child 8151 Jun 18 13:59:41 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:56 syslog0 last message repeated 15 timesJun 18 13:59:57 proxwww8155 Starting new child 8155 Jun 18 13:59:57 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:01 syslog0 last message repeated 4 timesJun 18 14:00:01 cron8177 (root) CMD (/usr/share/vzctl/scripts/vpsreboot)Jun 18 14:00:02 cron8181 (root) CMD (/usr/share/vzctl/scripts/vpsnetclean)Jun 18 14:00:02 cron8185 (root) CMD (test -x /usr/lib/atsar/atsa1 && /usr/lib/atsar/atsa1)Jun 18 14:00:02 cron8189 (root) CMD (vzdump --quiet --node 2 --snapshot --compress --dumpdir /tmp/windows --mailto office@netbone.ch --all)Jun 18 14:00:02 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:07 syslog0 last message repeated 5 timesJun 18 14:00:08 proxwww8231 Starting new child 8231 Jun 18 14:00:08 pvemirror25901 starting cluster syncronization Jun 18 14:00:08 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:09 pvemirror25901 syncing vzlist from '123.0.123.21' failed: 400 Invalid SOAPAction Jun 18 14:00:09 pvemirror25901 syncing templates Jun 18 14:00:09 pvemirror25901 cluster syncronization finished (0.58 seconds (files 0.00, config 0.00)) Jun 18 14:00:09 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:27 syslog0 last message repeated 18 timesJun 18 14:00:28 proxwww8233 Starting new child 8233 Jun 18 14:00:28 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:38 syslog0 last message repeated 10 timesJun 18 14:00:39 proxwww8250 Starting new child 8250 Jun 18 14:00:39 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:43 syslog0 last message repeated 4 timesJun 18 14:00:44 proxwww8253 Starting new child 8253 Jun 18 14:00:44 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:49 syslog0 last message repeated 5 timesJun 18 14:00:50 proxwww8254 Starting new child 8254 Jun 18 14:00:50 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:54 syslog0 last message repeated 4 timesJun 18 14:00:55 proxwww8257 Starting new child 8257 Jun 18 14:00:55 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:00 syslog0 last message repeated 5 timesJun 18 14:01:01 proxwww8271 Starting new child 8271 Jun 18 14:01:01 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:05 syslog0 last message repeated 4 timesJun 18 14:01:06 proxwww8275 Starting new child 8275 Jun 18 14:01:06 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:07 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:08 pvemirror25901 starting cluster syncronization Jun 18 14:01:08 pvemirror25901 syncing vzlist from '123.0.123.21' failed: 400 Invalid SOAPAction Jun 18 14:01:08 pvemirror25901 syncing templates Jun 18 14:01:08 pvemirror25901 cluster syncronization finished (0.52 seconds (files 0.00, config 0.00)) Jun 18 14:01:08 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:11 syslog0 last message repeated 3 timesJun 18 14:01:12 proxwww8276 Starting new child 8276 Jun 18 14:01:12 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:21 syslog0 last message repeated 9 timesJun 18 14:01:22 proxwww8257 unknown CID '7' Jun 18 14:01:22 proxwww8277 Starting new child 8277 Jun 18 14:01:22 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:23 proxwww8257 400 Invalid SOAPAction Jun 18 14:01:23 proxwww8282 Starting new child 8282 Jun 18 14:01:23 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:23 proxwww8284 Starting new child 8284 Jun 18 14:01:24 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:27 syslog0 last message repeated 3 timesJun 18 14:01:27 proxwww8275 unknown CID '7' Jun 18 14:01:28 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:28 proxwww8291 Starting new child 8291 Jun 18 14:01:29 proxwww8291 400 Invalid SOAPAction Jun 18 14:01:29 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:33 syslog0 last message repeated 4 timesJun 18 14:01:34 proxwww8282 unknown CID '7' Jun 18 14:01:34 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:34 proxwww8309 Starting new child 8309 Jun 18 14:01:35 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:35 proxwww8310 Starting new child 8310 Jun 18 14:01:35 proxwww8311 Starting new child 8311 Jun 18 14:01:36 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:39 syslog0 last message repeated 3 timesJun 18 14:01:39 proxwww8311 unknown CID '7' Jun 18 14:01:40 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:44 syslog0 last message repeated 4 timesJun 18 14:01:45 proxwww8311 unknown CID '7' Jun 18 14:01:45 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:47 syslog0 last message repeated 2 timesJun 18 14:01:47 proxwww8346 Starting new child 8346 Jun 18 14:01:48 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:49 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:50 proxwww8310 unknown CID '7' Jun 18 14:01:50 pvetunnel11768 trying to finish tunnel 18296
 
log on master:

Jun 18 13:59:07 syslog0 last message repeated 9 timesJun 18 13:59:08 pvemirror25901 starting cluster syncronization Jun 18 13:59:08 pvemirror25901 syncing vzlist from '123.0.123.21' failed: 400 Invalid SOAPAction Jun 18 13:59:08 pvemirror25901 syncing templates Jun 18 13:59:08 pvemirror25901 cluster syncronization finished (0.52 seconds (files 0.00, config 0.00)) Jun 18 13:59:08 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:13 syslog0 last message repeated 5 timesJun 18 13:59:14 proxwww8135 Starting new child 8135 Jun 18 13:59:14 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:19 syslog0 last message repeated 5 timesJun 18 13:59:20 proxwww8136 Starting new child 8136 Jun 18 13:59:20 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:25 syslog0 last message repeated 5 timesJun 18 13:59:26 proxwww8137 Starting new child 8137 Jun 18 13:59:26 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:40 syslog0 last message repeated 14 timesJun 18 13:59:41 proxwww8151 Starting new child 8151 Jun 18 13:59:41 pvetunnel11768 trying to finish tunnel 18296 Jun 18 13:59:56 syslog0 last message repeated 15 timesJun 18 13:59:57 proxwww8155 Starting new child 8155 Jun 18 13:59:57 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:01 syslog0 last message repeated 4 timesJun 18 14:00:01 cron8177 (root) CMD (/usr/share/vzctl/scripts/vpsreboot)Jun 18 14:00:02 cron8181 (root) CMD (/usr/share/vzctl/scripts/vpsnetclean)Jun 18 14:00:02 cron8185 (root) CMD (test -x /usr/lib/atsar/atsa1 && /usr/lib/atsar/atsa1)Jun 18 14:00:02 cron8189 (root) CMD (vzdump --quiet --node 2 --snapshot --compress --dumpdir /tmp/windows --mailto office@netbone.ch --all)Jun 18 14:00:02 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:07 syslog0 last message repeated 5 timesJun 18 14:00:08 proxwww8231 Starting new child 8231 Jun 18 14:00:08 pvemirror25901 starting cluster syncronization Jun 18 14:00:08 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:09 pvemirror25901 syncing vzlist from '123.0.123.21' failed: 400 Invalid SOAPAction Jun 18 14:00:09 pvemirror25901 syncing templates Jun 18 14:00:09 pvemirror25901 cluster syncronization finished (0.58 seconds (files 0.00, config 0.00)) Jun 18 14:00:09 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:27 syslog0 last message repeated 18 timesJun 18 14:00:28 proxwww8233 Starting new child 8233 Jun 18 14:00:28 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:38 syslog0 last message repeated 10 timesJun 18 14:00:39 proxwww8250 Starting new child 8250 Jun 18 14:00:39 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:43 syslog0 last message repeated 4 timesJun 18 14:00:44 proxwww8253 Starting new child 8253 Jun 18 14:00:44 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:49 syslog0 last message repeated 5 timesJun 18 14:00:50 proxwww8254 Starting new child 8254 Jun 18 14:00:50 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:00:54 syslog0 last message repeated 4 timesJun 18 14:00:55 proxwww8257 Starting new child 8257 Jun 18 14:00:55 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:00 syslog0 last message repeated 5 timesJun 18 14:01:01 proxwww8271 Starting new child 8271 Jun 18 14:01:01 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:05 syslog0 last message repeated 4 timesJun 18 14:01:06 proxwww8275 Starting new child 8275 Jun 18 14:01:06 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:07 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:08 pvemirror25901 starting cluster syncronization Jun 18 14:01:08 pvemirror25901 syncing vzlist from '123.0.123.21' failed: 400 Invalid SOAPAction Jun 18 14:01:08 pvemirror25901 syncing templates Jun 18 14:01:08 pvemirror25901 cluster syncronization finished (0.52 seconds (files 0.00, config 0.00)) Jun 18 14:01:08 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:11 syslog0 last message repeated 3 timesJun 18 14:01:12 proxwww8276 Starting new child 8276 Jun 18 14:01:12 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:21 syslog0 last message repeated 9 timesJun 18 14:01:22 proxwww8257 unknown CID '7' Jun 18 14:01:22 proxwww8277 Starting new child 8277 Jun 18 14:01:22 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:23 proxwww8257 400 Invalid SOAPAction Jun 18 14:01:23 proxwww8282 Starting new child 8282 Jun 18 14:01:23 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:23 proxwww8284 Starting new child 8284 Jun 18 14:01:24 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:27 syslog0 last message repeated 3 timesJun 18 14:01:27 proxwww8275 unknown CID '7' Jun 18 14:01:28 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:28 proxwww8291 Starting new child 8291 Jun 18 14:01:29 proxwww8291 400 Invalid SOAPAction Jun 18 14:01:29 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:33 syslog0 last message repeated 4 timesJun 18 14:01:34 proxwww8282 unknown CID '7' Jun 18 14:01:34 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:34 proxwww8309 Starting new child 8309 Jun 18 14:01:35 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:35 proxwww8310 Starting new child 8310 Jun 18 14:01:35 proxwww8311 Starting new child 8311 Jun 18 14:01:36 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:39 syslog0 last message repeated 3 timesJun 18 14:01:39 proxwww8311 unknown CID '7' Jun 18 14:01:40 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:44 syslog0 last message repeated 4 timesJun 18 14:01:45 proxwww8311 unknown CID '7' Jun 18 14:01:45 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:47 syslog0 last message repeated 2 timesJun 18 14:01:47 proxwww8346 Starting new child 8346 Jun 18 14:01:48 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:49 pvetunnel11768 trying to finish tunnel 18296 Jun 18 14:01:50 proxwww8310 unknown CID '7' Jun 18 14:01:50 pvetunnel11768 trying to finish tunnel 18296
 
on webpanel and cluster.cfg it is named node 8 and even after restart of clustersync and tunnel inside syslog i find 7 instead of 8:

Jun 18 14:32:42 stm01 postfix/qmgr[2752]: F2B27184223: removed
Jun 18 14:32:43 stm01 proxwww[12349]: Starting new child 12349
Jun 18 14:32:46 stm01 proxwww[12318]: unknown CID '7'
Jun 18 14:32:48 stm01 proxwww[12358]: Starting new child 12358
 
how did you re-install?

how did you remove the old (dead) node?
 
Old node was deleted in conf-file and removed ssh keys.

New install was done from scratch with latest updates and then apt-get update + upgrade + kernel update
 
i made clean install from sratch with latest iso and after that apt update + upgrade and then updated kernel to latest.

i removed by editing files .conf on master /pve and removed ssh keys.
 
i have now deleted the whole cluster configuration and made it new on all nodes:

CID----IPADDRESS----ROLE-STATE--------UPTIME---LOAD----MEM---ROOT---DATA
1 : 123.0.123.30 M S 23 days 02:20 0.51 60% 19% 86%
2 : 123.0.123.20 N S 262 days 21:40 0.66 71% 6% 16%
3 : 123.0.123.40 N S 23 days 01:50 0.05 79% 5% 84%
4 : 123.0.123.21 N ERROR: 400 Invalid SOAPAction
 
now made new server master - syslog shows:

Jun 18 17:51:34 cluster21 pvedaemon[4304]: WARNING: Cannot encode 'meminfo' element as 'hash'. Will be encoded as 'map' instead
Jun 18 17:51:35 cluster21 proxwww[7396]: no config for cluster node '8' at /usr/share/perl5/PVE/ConfigServer.pm line 1957.
Jun 18 17:51:36 cluster21 pvedaemon[4304]: WARNING: Cannot encode 'meminfo' element as 'hash'. Will be encoded as 'map' instead
Jun 18 17:51:40 cluster21 pvedaemon[7093]: WARNING: Cannot encode 'meminfo' element as 'hash'. Will be encoded as 'map' instead
Jun 18 17:51:41 cluster21 proxwww[7374]: no config for cluster node '8' at /usr/share/perl5/PVE/ConfigServer.pm line 1957.
Jun 18 17:51:42 cluster21 pvedaemon[4304]: WARNING: Cannot encode 'meminfo' element as 'hash'. Will be encoded as 'map' instead
Jun 18 17:51:46 cluster21 pvedaemon[4304]: WARNING: Cannot encode 'meminfo' element as 'hash'. Will be encoded as 'map' instead
Jun 18 17:51:46 cluster21 proxwww[7396]: no config for cluster node '8' at /usr/share/perl5/PVE/ConfigServer.pm line 1957.
Jun 18 17:51:48 cluster21 pvedaemon[4304]: WARNING: Cannot encode 'meminfo' element as 'hash'. Will be encoded as 'map' instead
 
Jun 18 17:51:35 cluster21 proxwww[7396]: no config for cluster node '8' at /usr/share/perl5/PVE/ConfigServer.pm line 1957.

Does 'pveca -l' work now? If so, try to remove "/var/lib/pve-manager/vzlist" and restart pvedaemon and apache2 (or reboot).