Remove node and add new one

sand-max

Well-Known Member
Apr 13, 2016
38
1
48
40
Hi guys!

I have 4 nodes in cluster but 4th node is dead. And I'd like remove node and replace one by new server but I found next notes:

"
Please be carefull, it a permanently remove !!!

  • Never restart the removed node
  • Don't assign the local ip address of the removed node to a new node
  • Never assign the name of the removed node to a new node
"

How to can I add 4th node safety with using name, IP old node? Or need remove 4 th node and add next 5th one?
 
  • Like
Reactions: G0ldmember
Hi,

Main problem of a node removal is a situation where the removed node comes up again and still thinks it's in the cluster (as it was kicked out it did not got notified about the kicking out, e.g., if it was offline during that.) as corosync.conf configuration files are still on the from-cluster-removed node.

If you install a new server and name it the same as the old, removed, one you can re-add it just fine.
Same IP is also not an issue, just ensure that the old node is really "dead", not that somebody starts it again and it tries to uses the now re-used IP (address conflicts)
 
  • Like
Reactions: G0ldmember
Hi,

Main problem of a node removal is a situation where the removed node comes up again and still thinks it's in the cluster (as it was kicked out it did not got notified about the kicking out, e.g., if it was offline during that.) as corosync.conf configuration files are still on the from-cluster-removed node.

If you install a new server and name it the same as the old, removed, one you can re-add it just fine.
Same IP is also not an issue, just ensure that the old node is really "dead", not that somebody starts it again and it tries to uses the now re-used IP (address conflicts)
Thanks!
I removed node4 "pvecm delnode node4" and rm -rf /etc/pve/nodes/node4 .
I see it's ok now via UI on node1,node3. But I see offline node4 on node2 via UI. I run pvecm status, and it's not one:
Code:
root@pvenode2:~# pvecm status
Quorum information
------------------
Date:             Thu Oct  3 12:57:03 2019
Quorum provider:  corosync_votequorum
Nodes:            3
Node ID:          0x00000002
Ring ID:          1/4432
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2 
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 10.10.3.201
0x00000002          1 10.10.3.202 (local)
0x00000003          1 10.10.3.203

I run pve5to6 and see next:
Code:
root@pvenode2:~# pve5to6 | grep FAIL
FAIL: 1 nodes are offline!
FAIL: corosync.conf (3) and pmxcfs (4) don't agree about size of nodelist.
FAIL: corosync 2.x installed, cluster-wide upgrade to 3.x needed!
FAILURES: 3
root@pvenode2:~#

And I found by "find" dir and file pvenode4 on all nodes:
Code:
root@pvenode2:~# find /  -name pvenode4
/var/lib/rrdcached/db/pve2-storage/pvenode4
/var/lib/rrdcached/db/pve2-node/pvenode4
root@pvenode2:~#
Do I need remove ones?
 
Do I need remove ones?

Not really necessary, only holds the old resource usage data from the removed node (historic CPU, Memory, ... use).

Hmm strange, can you post the output of
Code:
cat /etc/pve/.members
from that node 2?

Also try to restart the pve-cluster filesystem: systemctl restart pve-cluster corosync
 
Not really necessary, only holds the old resource usage data from the removed node (historic CPU, Memory, ... use).

Hmm strange, can you post the output of
Code:
cat /etc/pve/.members
from that node 2?

Also try to restart the pve-cluster filesystem: systemctl restart pve-cluster corosync


Thanks! After restart corosync it has gone
Code:
root@pvenode2:~# cat /etc/pve/.members
{
"nodename": "pvenode2",
"version": 16,
"cluster": { "name": "pvecluster1", "version": 7, "nodes": 4, "quorate": 1 },
"nodelist": {
  "pvenode1": { "id": 1, "online": 1, "ip": "10.10.3.201"},
  "pvenode2": { "id": 2, "online": 1, "ip": "10.10.3.202"},
  "pvenode3": { "id": 3, "online": 1, "ip": "10.10.3.203"},
  "pvenode4": { "id": 4, "online": 0}
  }
}
root@pvenode2:~# systemctl restart pve-cluster corosync
root@pvenode2:~# cat /etc/pve/.members
{
"nodename": "pvenode2",
"version": 0
}
root@pvenode2:~# cat /etc/pve/.members
{
"nodename": "pvenode2",
"version": 2,
"cluster": { "name": "pvecluster1", "version": 8, "nodes": 3, "quorate": 1 },
"nodelist": {
  "pvenode1": { "id": 1, "online": 1},
  "pvenode2": { "id": 2, "online": 1},
  "pvenode3": { "id": 3, "online": 1}
  }
}
root@pvenode2:~#
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!