[SOLVED] I dread every single time I have to touch any thing related to corosync

[ishanjain]

One of my node's IP that is part of the cluster has changed. I read a few threads here and they suggested changing the address in

1. /etc/hosts
2. /etc/network/interfaces(on the machine who's IP was changed)
3. /etc/corosync/corosync.conf

and then reboot.

I did all that and it still doesn't work. The UI still showed the old IP address for that node and I couldn't reach any thing on that node from the webui of a different node in the cluster.


Then I read more threads and found a suggestion that you also need to bump config_version. Okay, cool so I did that and rebooted everything again. Now, The status checkmark for that node in the web ui of a different node is green but I still can't reach anything on that node.

I dread and HATE every single time I have to do anything with corosync. The instructions and discussions on surrounding topics are all over the places and it always makes a mess. Every node in the cluster also has a v6 GUA. Why does it not use that at all ???

 

[waltar]

3. /etc/corosync/corosync.conf

Insrtad of "3)" do remove any vm/lxv ha configs, edit /etc/pve/corosync.conf (see path !!) and step up config number inside before press save (!!), systemctl restart corosync, push again ha configs if had before.

 

[ishanjain]

I do not have any VMs or LXC containers in HA. I edited /etc/pve/corosync.conf, bumped config_version from 4 to 5, restarted corosync. The config was synced to all the other nodes correctly but in the web ui I still have the same problem. i.e.

1. The check next to the node in the list on the left is green
2. In datacenter -> summary, I still see the old address for the node
3. I still can't access any of the data for that node from the web ui of a different node in the cluster.

And thank you for responding to my post so quickly!

 

[ishanjain]

Restarted pveproxy thinking that might solve the problem and it didn't.

 

[waltar]

Is ssh with new IP working and/or ssh to old one ?

 

[ishanjain]

ssh with new ip is working, ssh with old ip is not working.

 

[waltar]

I would reboot the pve node. Is the new IP in same network segment or outside the other nodes yet ?

 

[ishanjain]

The problematic node is `pvepc`. It's actual IP address is 10.0.99.4 but in the web ui it still shows 10.0.99.20.

In all corosync configs on all nodes, the ip address for pvepc is 10.0.99.4

 

[ishanjain]

> Is the new IP in same network segment or outside the other nodes yet ?

Same segment.
I have rebooted it 3-4 times now but I'll do it again.

 

[waltar]

Check again /etc/hosts for new IP entry after reboot (maybe pve rewrote it).

 

[ishanjain]

I don't know if pve added(or tried adding) any entries to /etc/hosts. I had added this entry manually in all nodes.

10.0.99.4 pvepc.home.arpa pvepc

After a reboot, I still have the same problem. Healthcheck passes but it still shows the old ip address and node is unreachable in the web ui

 

[waltar]

Funny ... still need some harder work to reach the goal. Did you thought about migrating everythink to remaining nodes, "de-join" the node from cluster, check again after new reboot for new ip in that then standalone node. If still don't want new IP reinstall it (15min) and join this "New" node.

 

[ishanjain]

This particular node (pvepc) is running vms with pci passthrough and I don't really want to migrate it to other nodes.

Dejoining a node from the cluster is hell and I don't want to do that if I can avoid it some how.

 

[waltar]

Yeah, I understand and feel same. Why not just stay then with old IP again ?? As all is in same segment it's only still a cosmetic problem if you want .4 instread of "life" with ".20".

 

[ishanjain]

I guess I can do that. I wanted a cleaner setup where all the proxmox nodes have addresses close to each other. .15 to .23 is now taken up by my wifi access points

 

[ishanjain]

I don't want to put it back on .20.

Please let me know if any one else knows whats going wrong here and how can i fix it, thanks! and thank you @waltar for teaching me how corosync updates are supposed to be done. Hopefully, Next time I will make less of a mess than I did now

 

[waltar]

What about entryi in /etc/issue and /etc/pve/.members following pveproxy restart ?

 

[ishanjain]

The /etc/issue entry on pvepc(machine with .20 address) shows 10.0.99.20

pvepc member in /etc/pve/.members on all machines points to 10.0.99.20

 

[waltar]

So edit these, restart pveproxy and what does the webui say ... ?!

 

[ishanjain]

/etc/pve/.members is readonly and I can't do chmod 600 on it even as a root user. Not sure how/what service is responsible for updating it