Remove node and add new one

[sand-max]

Hi guys!

I have 4 nodes in cluster but 4th node is dead. And I'd like remove node and replace one by new server but I found next notes:

"
Please be carefull, it a permanently remove !!!

• Never restart the removed node
• Don't assign the local ip address of the removed node to a new node
• Never assign the name of the removed node to a new node

"

How to can I add 4th node safety with using name, IP old node? Or need remove 4 th node and add next 5th one?

 

[t.lamprecht]

Hi,

Main problem of a node removal is a situation where the removed node comes up again and still thinks it's in the cluster (as it was kicked out it did not got notified about the kicking out, e.g., if it was offline during that.) as corosync.conf configuration files are still on the from-cluster-removed node.

If you install a new server and name it the same as the old, removed, one you can re-add it just fine.
Same IP is also not an issue, just ensure that the old node is really "dead", not that somebody starts it again and it tries to uses the now re-used IP (address conflicts)

 

[t.lamprecht]

Also where did you found those notes they do not seem really true, or at least not in a general way.

 

[sand-max]

Hi,

Main problem of a node removal is a situation where the removed node comes up again and still thinks it's in the cluster (as it was kicked out it did not got notified about the kicking out, e.g., if it was offline during that.) as corosync.conf configuration files are still on the from-cluster-removed node.

If you install a new server and name it the same as the old, removed, one you can re-add it just fine.
Same IP is also not an issue, just ensure that the old node is really "dead", not that somebody starts it again and it tries to uses the now re-used IP (address conflicts)

Thanks!
I removed node4 "pvecm delnode node4" and rm -rf /etc/pve/nodes/node4 .
I see it's ok now via UI on node1,node3. But I see offline node4 on node2 via UI. I run pvecm status, and it's not one:

root@pvenode2:~# pvecm status
Quorum information
------------------
Date:             Thu Oct  3 12:57:03 2019
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000002
Ring ID:          1/4432
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.10.3.201
0x00000002          1 10.10.3.202 (local)
0x00000003          1 10.10.3.203

I run pve5to6 and see next:

root@pvenode2:~# pve5to6 | grep FAIL
FAIL: 1 nodes are offline!
FAIL: corosync.conf (3) and pmxcfs (4) don't agree about size of nodelist.
FAIL: corosync 2.x installed, cluster-wide upgrade to 3.x needed!
FAILURES: 3
root@pvenode2:~#

And I found by "find" dir and file pvenode4 on all nodes:

root@pvenode2:~# find /  -name pvenode4
/var/lib/rrdcached/db/pve2-storage/pvenode4
/var/lib/rrdcached/db/pve2-node/pvenode4
root@pvenode2:~#

Do I need remove ones?

 

[t.lamprecht]

Do I need remove ones?

Not really necessary, only holds the old resource usage data from the removed node (historic CPU, Memory, ... use).

Hmm strange, can you post the output of

cat /etc/pve/.members

from that node 2?

Also try to restart the pve-cluster filesystem: systemctl restart pve-cluster corosync

 

[sand-max]

Not really necessary, only holds the old resource usage data from the removed node (historic CPU, Memory, ... use).

Hmm strange, can you post the output of

cat /etc/pve/.members

from that node 2?

Also try to restart the pve-cluster filesystem: systemctl restart pve-cluster corosync

Thanks! After restart corosync it has gone

root@pvenode2:~# cat /etc/pve/.members
{
"nodename": "pvenode2",
"version": 16,
"cluster": { "name": "pvecluster1", "version": 7, "nodes": 4, "quorate": 1 },
"nodelist": {
  "pvenode1": { "id": 1, "online": 1, "ip": "10.10.3.201"},
  "pvenode2": { "id": 2, "online": 1, "ip": "10.10.3.202"},
  "pvenode3": { "id": 3, "online": 1, "ip": "10.10.3.203"},
  "pvenode4": { "id": 4, "online": 0}
  }
}
root@pvenode2:~# systemctl restart pve-cluster corosync
root@pvenode2:~# cat /etc/pve/.members
{
"nodename": "pvenode2",
"version": 0
}
root@pvenode2:~# cat /etc/pve/.members
{
"nodename": "pvenode2",
"version": 2,
"cluster": { "name": "pvecluster1", "version": 8, "nodes": 3, "quorate": 1 },
"nodelist": {
  "pvenode1": { "id": 1, "online": 1},
  "pvenode2": { "id": 2, "online": 1},
  "pvenode3": { "id": 3, "online": 1}
  }
}
root@pvenode2:~#

 

[t.lamprecht]

great to hear

 

[sand-max]

Also need clean up ssh_keys of old node from file /etc/ssh/ssh_known_hosts