Simple node re-install creates chaos?

[LiveFreeOrPi]

I've been testing a simple two node cluster for a couple of months now, before considering a production deployment. I removed one node today and reloaded it with a new image, making tweaks to the previous disk partitions, design, etc. In any event, it has a new name, but the same IP address. What a mistake! The host key verification BUG is real. Shell access failed but was easy to repair, migrations fail hard.

So, fine, again, a test bed, so I'll simply change the IP address. Mistake number two! The GUI IP address change modifies /etc/network/interfaces, so why doesn't it change /etc/pve/corosync.conf? Why does the data center cluster insist on reflecting the old IP address? Where else to I have to hunt for config files to change?

This is from the documentation "Editing the corosync.conf file is not always very straightforward. There are two on each cluster node, one in /etc/pve/corosync.conf and the other in /etc/corosync/corosync.conf. Editing the one in our cluster file system will propagate the changes to the local one, but not vice versa."

I am super frustrated after down VMs and a rabbit hole of problems for what should be simple changes, so I apologize that my first post here is a bad user experience, but I just don't see how this software is "production ready". There seems to be a fairly robust community, but also a fair amount of open or long standing unresolved bugs https://bugzilla.proxmox.com/show_bug.cgi?id=4252

 

[UdoB]

I am super frustrated

Yes, learning and handling complex systems can have a lot more pitfalls than one expects.

When I tried to run an alternative cluster-capable virtualizer (not the Elephant in the room) I had experienced similar problems.

There is no way around it, except... learning.

In any way: welcome to the club ;-)

 

[zzz09700]

For cluster sized system its hard to detect (and handle) a server os reinstall.
A better approach would be set the server up as another brand new node, add it to the cluster and then remove signs of old server within the cluster.

 

[LiveFreeOrPi]

Yes, learning and handling complex systems can have a lot more pitfalls than one expects.

When I tried to run an alternative cluster-capable virtualizer (not the Elephant in the room) I had experienced similar problems.

There is no way around it, except... learning.

In any way: welcome to the club ;-)

I would have liked a more helpful response as to how to fix the simple issue of reusing an IP address in Proxmox rather than the condescending reply. I have been in the industry for decades-- learning is not new to me and I quite enjoy it. As I noted, some of the open issues are a year or more old which makes chasing them down more difficult than it should be.

 

[LiveFreeOrPi]

For cluster sized system its hard to detect (and handle) a server os reinstall.
A better approach would be set the server up as another brand new node, add it to the cluster and then remove signs of old server within the cluster.

As I mentioned I did install a completely new node (with a different name), I simply re-used the IP address. I do understand that the underlying SSH key is the root cause, but this software seems to handle it poorly.

 

[UdoB]

I would have liked a more helpful response as to how to fix the simple issue of reusing an IP address in Proxmox rather than the condescending reply.

Please excuse me if you found it to be condescending, that was not my intention.

Read your first post: there is no way I could recognize you as someone with decades of experience instead of being a freshman. I just wanted to send you a "welcome!".

:-(

 

[bbgeek17]

Read your first post: there is no way I could recognize you as someone with decades of experience instead of being a freshman.

To be frank, that was my impression as well.

I am super frustrated

It happens to everyone at some point in this field.

That said, you have not explained how you actually got there. Did you follow the documented procedure for removing "failed" node before re-adding it?

Since you are not in production yet, my advice would be : Start from scratch. If node replacement is your goal - repeat it by following official procedure. Record all steps, so that what you did is repeatable. If you get into "bad" situation again, list the steps with brief technical explanation, including command line outputs.

Good luck.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[LiveFreeOrPi]

For background purposes, I, like many, am evaluating replacements for VMWare. I've used ESXi at home for years, but have also used Citrix (and Xen), Hyper-V, Docker and we maintain a very expensive VCenter environment at work, where the licensing expires next March. Tick tock.

But back to my actual question-- The GUI IP address change modifies /etc/network/interfaces, so why doesn't it change /etc/pve/corosync.conf? Why does the data center cluster insist on reflecting the old IP address? Where else to I have to hunt for config files to change?

Doing this via the GUI to a node in a cluster breaks things. There are at least 3 options that I think could be helpful-

a. A pop up a screen indicating "are you sure you want to do change this node's IP address?" and perhaps point you to some documentation.
b. Inform the user with some language like "this GUI change does not change corosync.conf or update host files, do so manually, here's how:"
c. Remove the change IP address option from the GUI of a cluster until it works properly.

Changing hypervisors is obviously not a project that should be taken lightly, and threads like the SSH known_hosts bug(s) has me worrying about choosing Proxmox, regardless of the end user's level of expertise. https://forum.proxmox.com/threads/s...ass-ssh-known_hosts-bug-s.137809/#post-614017