2 node host swapping

L

Lost_My_Ones_and_Zeros

New Member
Jun 5, 2025
14
1
3
Easy question yet has bit me in the arse more times then I care to admit. I know it's said that proxmox is not dependent on a single node for a master and everything can run on any but this is the assumption that 3 or more are in use.

I need to swap 2 nodes out in a 2 node cluster for a little more beefy boys. Not enough physical or switch room to do have all running so it's the VM shuffle game.

pvecm status
2 nodes active along with qdevice.
No HA enabled on VM's - makes this easy in theory.

Evac all VM's from the first node to be removed.
systemctl stop pve-cluster
systemctl stop corosync

Move to the one holding all the VM's
Run
pvecm delnode "name of host to remove"
pvecm nodes "verify node is gone"

My concern and I've had this happen a lot over the years is the remaining one is still active but when you click on Cluster Information and Join Information the IP from the other one is still listed. Well that's useless as I can't join a 2nd node back if that IP doesn't change. How to manually update that IP or am I just screwed when that IP holder is dead. Honest question as I've lost a few dev clusters to this very issue.
 
I think it would be easier to just make a new cluster. I'm picturing steps like this:
Move all VMs to one node, and decommission the other. Set up new one, and use the datacenter manager to move the VMs from the old one to the new one. Replace the old node with the second new one. Install the other new one and set up a new cluster with the two new nodes.
 
Shutting down a node and/or disconnecting it from corosync in a 2-node cluster will block changes on the remaining node, if there is no qdevice running too.
If there is no shared storage, firewall config, sdn etc to move, creating a new cluster with the new hardware will be a clean independent approach. Instead of using PDM as @tomservo suggested, I would simply do a qm remote-migrate for the vms.
If you want to go the node-switching path: Evac all VMs from the first node to be removed, shutdown this node. On the remaining one in a shell (preferably via ssh or local console, not via webui) do a killall pmxcfs, then pmxcfs -l, then edit /etc/pve/corosync.conf and remove the entry for the decommissioned node and increment the config_version. Run killall pmxcfs again and then systemctl restart pve-cluster corosync. Remove /etc/pve/nodes/<oldnode> folder. Then join the new node. Repeat for second node replacement.
I haven't tested these steps, btw.
 
  • Like
Reactions: Johannes S
Think I agree with @tomservo on this one. Granted won't use datacenter manger I haven't found a use for it on small clusters. Honestly haven't touched it since release so no idea what's new in it. Until it can show each vm disk latency, io, throughtput per machine i have no use for it. Better off with outside monitoring still imho.

@Johannes S Yes one thing. The main issue I've run into almost every time is the main cluster join IP never changes from the original nodes that cluster was started on even after following the steps for a clean remove. Would be nice if I could change that IP manually before hand as removing the node that holds that IP always ends badly for me. Mind you I haven't done this in v9 yet but history has a way of leaving scars you don't forget.

I'll shut down do full backup then restore to a new cluster. Less bs to deal with on small clusters at least. I lost all faith in proxomox using qdevices on doing node swaps on sub 3 real node units. Things just get to weird to deal with corosync becomes to angry to deal with and just fails resulting in full rebuild.
 
Confirmed that join IP does change at least on 9.2.3. Went back to 8.1 which I know needs to be updated and it did not change between the two nodes that was after Crtl+F5 and clear browser cache marked for all time.
 
You must log in or register to reply here.
Top Bottom
Back