[SOLVED] Change the IP address of one node inside a cluster

Mephisto aD · Jun 1, 2021

Hi there,

I'm recently having a two nodes Cluster of PVE in my home environment.
Caused by some accidents i needed to change the IP address of one node (the other one remained where it was on the IP-range)

What did i do?

On both nodes (while they still had quorum) i changed the corosync.conf running :

Bash:

nano /etc/pve/corosync.conf

and finally restarting the service on both nodes (just in case ;-)

Bash:

systemctl restart corosync

after that i needed to change the nodes IP-address:

Bash:

nano /etc/hosts
nano /etc/network/interfaces
systemctl restart networking

[This is the point where I am right now]

So now i have two issues which seem to be leftovers from the IP-change:

1. When I now move anything to the node with the new IP I receive this output in the Migration window:

Code:

2021-06-01 11:38:36 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=<NODE_WITH_NEW_IP>' root@<THE_NODES_OLD_IP> /bin/true
2021-06-01 11:38:36 ssh: connect to host <THE_NODES_OLD_IP> port 22: No route to host
2021-06-01 11:38:36 ERROR: migration aborted (duration 00:00:03): Can't connect to destination address using public key
TASK ERROR: migration aborted

If i get this correctly for the migrations there must be another config file storing the IP-adresses of the other nodes. Does anyone know where I need to edit this?

2. Logging into the webinterface on the node which didn't change I usually also saw the stats of the VMs from the other node (where I changed the IP-address). Unfortunately this looks like that now:

<description for those who aren't logged in here: Loading bar which desn't change any more, Message: "Error Connection error: 595: No route to host">

On the web-ui of the node with changed address anyway this doesn't occur and i cann see all the stats from both nodes.

So maybe in this second case it also tries to still connect to the old address of this node? How can fix this?

Thanks a lot already

This forum with all the help i found here reading in your threads is simply great!

oguz · Jun 1, 2021

hi,

maphisto_ad said:
What did i do?

On both nodes (while they still had quorum) i changed the corosync.conf running :

Bash:

nano /etc/pve/corosync.conf

don't edit this file directly. please see the instructions here [0] and here [1]

If your node is in a cluster, where it is not recommended to change its name, adapt /etc/pve/corosync.conf so that the nodes name is changed also there. See [0]

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_edit_corosync_conf
[1]: https://pve.proxmox.com/wiki/Renaming_a_PVE_node

Mephisto aD · Jun 1, 2021

oguz said:
don't edit this file directly. please see the instructions here [0] and here [1]

[0]: https://pve.proxmox.com/pve-docs/chapter-pvecm.html#pvecm_edit_corosync_conf
[1]: https://pve.proxmox.com/wiki/Renaming_a_PVE_node

hi Oguz,

I just checked the two files /etc/pve/corosync.conf and /etc/corosync/corosync.conf on both nodes again and they don't seem to be different, neither they have any mistakes (as I can see). All files are synced and the cluster has quorum again. So it seems to work, apart from the described exceptions.

are there any more files where the nodes ips are stored?

Thanks a lot for your quick answer
maphisto

oguz · Jun 1, 2021

please post the outputs of the following commands:

Code:

pvecm status
cat /etc/hosts
cat /etc/pve/corosync.conf

Mephisto aD · Jun 1, 2021

Well, hopefully I just missed something

Code:

$ pvecm status

Cluster information
-------------------
Name:             JUPITER
Config Version:   3
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Tue Jun  1 12:54:13 2021
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.ec
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          2 192.168.2.10 (local)
0x00000002          1 192.168.2.3

Code:

$ cat /etc/hosts

127.0.0.1 localhost.localdomain localhost
192.168.2.10 GANYMED.local GANYMED
192.168.2.3 CALLISTO.local CALLISTO

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

I made the /etc/hosts similar on both nodes as i first thought it would maybe help anything.

Code:

$ cat /etc/pve/corosync.conf

logging {
  debug: off
  to_syslog: yes
}

nodelist {
  node {
    name: CALLISTO
    nodeid: 2
    quorum_votes: 1
    ring0_addr: 192.168.2.3
  }
  node {
    name: GANYMED
    nodeid: 1
    quorum_votes: 2
    ring0_addr: 192.168.2.10
  }
}

quorum {
  provider: corosync_votequorum
}

totem {
  cluster_name: JUPITER
  config_version: 3
  interface {
    linknumber: 0
  }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
  version: 2
}

Some notes about this:
GANYMED is the node which changed the IP from 192.168.2.2 to 192.168.2.10
CALLISTO [on 192.168.2.3] didn't change nothing

oguz · Jun 1, 2021

so if you run ssh 192.168.2.10 from 192.168.2.3 does it login as root? (and vice versa?)

do you see any error messages in journalctl?

Mephisto aD · Jun 1, 2021

oguz said:
so if you run ssh 192.168.2.10 from 192.168.2.3 does it login as root? (and vice versa?)

yes it does in both ways

oguz said:
do you see any error messages in journalctl?

no, at least no messages from today

is this a good or a bad thing?

oguz · Jun 1, 2021

maphisto_ad said:
yes it does in both ways

okay

can you also post the output of cat /etc/pve/.members from both nodes?

you can also try restarting corosync and pve-cluster services.

also check if ganymed resolved to the new IP address you changed to.

Mephisto aD · Jun 1, 2021

hmm, I guess this was the golden hint:

Code:

root@CALLISTO:~# cat /etc/pve/.members
{
"nodename": "CALLISTO",
"version": 6,
"cluster": { "name": "JUPITER", "version": 3, "nodes": 2, "quorate": 1 },
"nodelist": {
  "CALLISTO": { "id": 2, "online": 1, "ip": "192.168.2.3"},
  "GANYMED": { "id": 1, "online": 1, "ip": "192.168.2.2"}
  }
}

Code:

root@GANYMED:~# cat /etc/pve/.members
{
"nodename": "GANYMED",
"version": 5,
"cluster": { "name": "JUPITER", "version": 3, "nodes": 2, "quorate": 1 },
"nodelist": {
  "CALLISTO": { "id": 2, "online": 1, "ip": "192.168.2.3"},
  "GANYMED": { "id": 1, "online": 1, "ip": "192.168.2.2"}
  }
}

this files aren't changed yet.

but when I try to adjust this file with nano i get this error:

<description for those who aren't logged in here: can't exit the editor saving the changes "Error writing /etc/pve/.members: Input/output error">

is there anything i need to pay attention to before changing the files content?

oguz · Jun 1, 2021

you haven't restarted pve-cluster service. after restart it should be regenerated

Mephisto aD · Jun 1, 2021

oguz said:
you haven't restarted pve-cluster service. after restart it should be regenerated

oh, well you seem to be right with that
after

Code:

systemctl restart pve-cluster && systemctl restart corosync

on both nodes my issues seem to be gone

Thanks a lot for your patience
Best wishes
maphisto

oguz · Jun 2, 2021

great, you're welcome

please mark the thread as [SOLVED] so others also know what to expect

Search

Search

[SOLVED] Change the IP address of one node inside a cluster

Mephisto aD

New Member

oguz

Proxmox Retired Staff

Mephisto aD

New Member

oguz

Proxmox Retired Staff

Mephisto aD

New Member

oguz

Proxmox Retired Staff

Mephisto aD

New Member

oguz

Proxmox Retired Staff

Mephisto aD

New Member

oguz

Proxmox Retired Staff

Mephisto aD

New Member

oguz

Proxmox Retired Staff