[SOLVED] Change the IP address of one node inside a cluster

Mephisto aD

New Member
Jun 1, 2021
12
2
3
22
Hi there,

I'm recently having a two nodes Cluster of PVE in my home environment.
Caused by some accidents i needed to change the IP address of one node (the other one remained where it was on the IP-range)

What did i do?

On both nodes (while they still had quorum) i changed the corosync.conf running :

Bash:
nano /etc/pve/corosync.conf

and finally restarting the service on both nodes (just in case ;-)

Bash:
systemctl restart corosync

after that i needed to change the nodes IP-address:

Bash:
nano /etc/hosts
nano /etc/network/interfaces
systemctl restart networking

[This is the point where I am right now]

So now i have two issues which seem to be leftovers from the IP-change:

1. When I now move anything to the node with the new IP I receive this output in the Migration window:

Code:
2021-06-01 11:38:36 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=<NODE_WITH_NEW_IP>' root@<THE_NODES_OLD_IP> /bin/true
2021-06-01 11:38:36 ssh: connect to host <THE_NODES_OLD_IP> port 22: No route to host
2021-06-01 11:38:36 ERROR: migration aborted (duration 00:00:03): Can't connect to destination address using public key
TASK ERROR: migration aborted

If i get this correctly for the migrations there must be another config file storing the IP-adresses of the other nodes. Does anyone know where I need to edit this?

2. Logging into the webinterface on the node which didn't change I usually also saw the stats of the VMs from the other node (where I changed the IP-address). Unfortunately this looks like that now:

1622540780881.png
<description for those who aren't logged in here: Loading bar which desn't change any more, Message: "Error Connection error: 595: No route to host">

On the web-ui of the node with changed address anyway this doesn't occur and i cann see all the stats from both nodes.


So maybe in this second case it also tries to still connect to the old address of this node? How can fix this?

Thanks a lot already :)
This forum with all the help i found here reading in your threads is simply great!
 

oguz

Proxmox Staff Member
Retired Staff
Nov 19, 2018
5,207
680
118
hi,

What did i do?

On both nodes (while they still had quorum) i changed the corosync.conf running :

Bash:
nano /etc/pve/corosync.conf
don't edit this file directly. please see the instructions here [0] and here [1]

If your node is in a cluster, where it is not recommended to change its name, adapt /etc/pve/corosync.conf so that the nodes name is changed also there. See [0]

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_edit_corosync_conf
[1]: https://pve.proxmox.com/wiki/Renaming_a_PVE_node
 

Mephisto aD

New Member
Jun 1, 2021
12
2
3
22
don't edit this file directly. please see the instructions here [0] and here [1]

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_edit_corosync_conf
[1]: https://pve.proxmox.com/wiki/Renaming_a_PVE_node
hi Oguz,

I just checked the two files /etc/pve/corosync.conf and /etc/corosync/corosync.conf on both nodes again and they don't seem to be different, neither they have any mistakes (as I can see). All files are synced and the cluster has quorum again. So it seems to work, apart from the described exceptions.

are there any more files where the nodes ips are stored?

Thanks a lot for your quick answer
maphisto
 

oguz

Proxmox Staff Member
Retired Staff
Nov 19, 2018
5,207
680
118
please post the outputs of the following commands:
Code:
pvecm status
cat /etc/hosts
cat /etc/pve/corosync.conf
 

Mephisto aD

New Member
Jun 1, 2021
12
2
3
22
Well, hopefully I just missed something :)

Code:
$ pvecm status

Cluster information
-------------------
Name:             JUPITER
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  1 12:54:13 2021
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.ec
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 192.168.2.10 (local)
0x00000002          1 192.168.2.3

Code:
$ cat /etc/hosts

127.0.0.1 localhost.localdomain localhost
192.168.2.10 GANYMED.local GANYMED
192.168.2.3 CALLISTO.local CALLISTO

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I made the /etc/hosts similar on both nodes as i first thought it would maybe help anything.

Code:
$ cat /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: CALLISTO
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.2.3
  }
  node {
    name: GANYMED
    nodeid: 1
    quorum_votes: 2
    ring0_addr: 192.168.2.10
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: JUPITER
  config_version: 3
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Some notes about this:
GANYMED is the node which changed the IP from 192.168.2.2 to 192.168.2.10
CALLISTO [on 192.168.2.3] didn't change nothing
 
Last edited:

oguz

Proxmox Staff Member
Retired Staff
Nov 19, 2018
5,207
680
118
so if you run ssh 192.168.2.10 from 192.168.2.3 does it login as root? (and vice versa?)

do you see any error messages in journalctl?
 

oguz

Proxmox Staff Member
Retired Staff
Nov 19, 2018
5,207
680
118
yes it does in both ways
okay

can you also post the output of cat /etc/pve/.members from both nodes?

you can also try restarting corosync and pve-cluster services.

also check if ganymed resolved to the new IP address you changed to.
 

Mephisto aD

New Member
Jun 1, 2021
12
2
3
22
hmm, I guess this was the golden hint:

Code:
root@CALLISTO:~# cat /etc/pve/.members
{
"nodename": "CALLISTO",
"version": 6,
"cluster": { "name": "JUPITER", "version": 3, "nodes": 2, "quorate": 1 },
"nodelist": {
  "CALLISTO": { "id": 2, "online": 1, "ip": "192.168.2.3"},
  "GANYMED": { "id": 1, "online": 1, "ip": "192.168.2.2"}
  }
}

Code:
root@GANYMED:~# cat /etc/pve/.members
{
"nodename": "GANYMED",
"version": 5,
"cluster": { "name": "JUPITER", "version": 3, "nodes": 2, "quorate": 1 },
"nodelist": {
  "CALLISTO": { "id": 2, "online": 1, "ip": "192.168.2.3"},
  "GANYMED": { "id": 1, "online": 1, "ip": "192.168.2.2"}
  }
}

this files aren't changed yet.

but when I try to adjust this file with nano i get this error:

1622552311349.png
<description for those who aren't logged in here: can't exit the editor saving the changes "Error writing /etc/pve/.members: Input/output error">

is there anything i need to pay attention to before changing the files content?
 

oguz

Proxmox Staff Member
Retired Staff
Nov 19, 2018
5,207
680
118
you haven't restarted pve-cluster service. after restart it should be regenerated
 

Mephisto aD

New Member
Jun 1, 2021
12
2
3
22
you haven't restarted pve-cluster service. after restart it should be regenerated
oh, well you seem to be right with that
after
Code:
systemctl restart pve-cluster && systemctl restart corosync
on both nodes my issues seem to be gone

Thanks a lot for your patience
Best wishes
maphisto
 

oguz

Proxmox Staff Member
Retired Staff
Nov 19, 2018
5,207
680
118
great, you're welcome :)

please mark the thread as [SOLVED] so others also know what to expect
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!