How to properly separate two nodes from a cluster ?

H

hugosxm

Member
Jul 23, 2014
46
0
6
Montpellier, France
Hi proxmox forum,

I have always found good help here so i am asking again :)

I have a cluster with two nodes, qorum set to 1, tag "two nodes" in cluster.conf, etc... working like a charm ! :D

proxmox and proxmox2 are my two nodes, but proxmox2 have to move to my school, so i have to "separate" the two nodes WITHOUT reinstalling it, i want to keep the VM on each...

If i delete proxmox2 from the cluster, proxmox will stay the only one in the cluster, is this ok ? Or i need to remove BOTH node from the cluster ?!


And how to remove a node properly ? i mean wihout reinstalling proxmox from scratch ? i know that this is not a good option in real life but this is not production here... Just a school project...

i was thinking to do the following

remove proxmox2 from cluster then run :

service cman stop

killall -9 corosync cman dlm_controld fenced

service pve-cluster stop

rm /etc/cluster/cluster.conf

rm -rf /var/lib/pve-cluster/* /var/lib/pve-cluster/.*

rm /var/lib/cluster/*

and reboot i guess


proxmox2 will never be in the cluster again after this...


Please give me your opinion on that :)

thank you

PS : Everybody is running proxmox for the project in my classroom because we really pushed for this with a friend :cool:
 
Hi,
the easiest way is connect on one node then type
Code: 
pvecm delnode <outhernode>
pvecm delnode <onnode>
reboot
 
Forget this it is not working on the old cluster.
sorry
 
This is not working on the last stable version!
 
your version should work, but you have to save your Vm config and storage, because you make a full reset of the config.
You must copy them back but the rest is ok.
and do the same on the outher node.
 
Last edited:
OK so I back up etc/pve/storage
And etc/pve/qemu-server

On both node

Then I remove both node from cluster with my "method"

Then reboot

Then put back the file in etc/pve

That's right?
 
Hi,

Just to mention, your process you are trying to confirm looks very similar to the one discussed here:
http://undefinederror.org/how-to-reset-cluster-configuration-in-proxmox-2/


FYI I had to do this yesterday when stripping off and re-doing cluster config on a failed cluster build config attempt.

The steps detailed at the URL above appeared to work fine. However, caveat to mention, I had no VMs in my cluster and nothing to lose. So no big complexity for me with the 'backup and restore' :)

but based on the thread here, and the suggestions at link above, I believe it is the case that this process should work. You just need to backup your config files betore burning the cluster config directories/structure to the ground; then once you get hosts back up as standalone hosts, you can restore the proper config files so that the proxmox host is aware that it does in fact have some VMs defined locally that it should allow you to use.

Maybe good idea if you are super paraniod to have backups / dumps of your VMs first :)

But I think you should be fine following the hints at the URL http://undefinederror.org/how-to-reset-cluster-configuration-in-proxmox-2/

which are not dissimilar from what you planned to do already.

Good luck!


Tim
 
I did what i wrote and backup the file after the "cleaning", all is working fine on the two nodes :)

The only thing you have to rebuild "by hand" in the Web UI is the vzdump backup task ...

But on ONE node i have a problem now, the console is not working and i've got the following in my logs :

Code: 
Jun 19 13:00:20 proxmox pveproxy[5174]: proxy detected vanished client connection
Jun 19 13:00:20 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: ssl handshake failure
Jun 19 13:00:21 proxmox pveproxy[5174]: proxy detected vanished client connection
Jun 19 13:00:23 proxmox pveproxy[5173]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:23 proxmox pveproxy[5173]: Can't call method "timeout_reset" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 225.
Jun 19 13:00:23 proxmox pveproxy[5175]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:24 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:24 proxmox pveproxy[5175]: proxy detected vanished client connection
Jun 19 13:00:24 proxmox pveproxy[5174]: proxy detected vanished client connection
Jun 19 13:00:27 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:27 proxmox pveproxy[5174]: Can't call method "timeout_reset" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 225.
Jun 19 13:00:29 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:30 proxmox pveproxy[5174]: proxy detected vanished client connection
Jun 19 13:00:31 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:32 proxmox pveproxy[5174]: proxy detected vanished client connection
Jun 19 13:00:33 proxmox pveproxy[5173]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:33 proxmox pveproxy[5173]: Can't call method "timeout_reset" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 225.
Jun 19 13:00:33 proxmox pveproxy[5175]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:34 proxmox pveproxy[5175]: proxy detected vanished client connection
Jun 19 13:00:37 proxmox pveproxy[5175]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:37 proxmox pveproxy[5175]: Can't call method "timeout_reset" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 225.
Jun 19 13:00:37 proxmox pveproxy[5175]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:38 proxmox pveproxy[5175]: proxy detected vanished client connection
Jun 19 13:00:46 proxmox pveproxy[5175]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:46 proxmox pveproxy[5175]: proxy detected vanished client connection
Jun 19 13:00:47 proxmox pveproxy[5173]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:47 proxmox pveproxy[5173]: Can't call method "timeout_reset" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 225.
Jun 19 13:00:47 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:48 proxmox pveproxy[5174]: proxy detected vanished client connection
Jun 19 13:00:49 proxmox pveproxy[5174]: problem with client 172.20.50.55; ssl3_read_bytes: tlsv1 alert unknown ca
Jun 19 13:00:49 proxmox pveproxy[5174]: Can't call method "timeout_reset" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 225.

i tried this :

pvecm updatecerts
clear browser cache
/etc/init.d/pveproxy restart
/etc/init.d/pvedaemon restart

same problem..

then i tried with " --force" and "-force"

same problem...

so i manually deleted the file in /etc/pve/ by hand then do a pvecm updatecerts

probleme here again :(

I also checked the time, ntp seems to be ok ...

Do i have to open another thread for this maybe ?
 
Last edited:
Hi, for what it is worth, the specific words in the error message you post, "tlsv1 alert unknown <caJun>" I think are a hint here - it looks like there is a TLS Cert or some kind of trust relationship which is now broken.

I'm assuming you have ?
- stripped old cluster config
- rebooted
- rebuilt fresh cluster config
- can now ssh from first node to second node, and no root pass is prompted/required to do so ?
- can manage both nodes from web GUI ? (can you manage tasks on second host while on the first host web admin panel for example, ie, delegate create new VM on Prox2 host while logged in to webUI for Prox1 host ?)

if the last 2 points above are not true, it suggests to me the cluster is not in place correctly / that the hosts don't trust each other / and it is broken.

just a thought / guess though.


Tim
 
thanks for the answer :)

there is no more cluster right now, the two server are in two different place and are not working together anymore
i did not make a new fresh cluster config, just a standalone config...
 
I am still searching what the problem with that SSL error

I tried that :

Code: 
root /etc/pve # update-ca-certificates --fresh
Clearing symlinks in /etc/ssl/certs...done.
Updating certificates in /etc/ssl/certs... 171 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.

But same error :(

Any help will be great ! thanks !

EDIT !

now i can't even more log to the web ui, it says : "server online ?"

then logs are like this now :

Code: 
Jun 21 00:22:51 proxmox pveproxy[3170]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulusJun 21 00:22:51 proxmox pveproxy[3169]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:51 proxmox pveproxy[3169]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:54 proxmox pveproxy[3170]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:54 proxmox pveproxy[3170]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:54 proxmox pveproxy[3169]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:54 proxmox pveproxy[3169]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:57 proxmox pveproxy[3170]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:57 proxmox pveproxy[3169]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:57 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:57 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:00 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:00 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:01 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:01 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:04 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:04 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:05 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:05 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus


EDIT 2

OK i am pist **f right now... so i will change my plan : if i backup my vm conf, storage.cfg, etc... then i do a pve reinstall by aptitude, then put the conf back in pve, can it solve the problem ?
 
Last edited:
hugosxm said:
I am still searching what the problem with that SSL error

I tried that :

Code: 
root /etc/pve # update-ca-certificates --fresh
Clearing symlinks in /etc/ssl/certs...done.
Updating certificates in /etc/ssl/certs... 171 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....done.

But same error :(

Any help will be great ! thanks !

EDIT !

now i can't even more log to the web ui, it says : "server online ?"

then logs are like this now :

Code: 
Jun 21 00:22:51 proxmox pveproxy[3170]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulusJun 21 00:22:51 proxmox pveproxy[3169]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:51 proxmox pveproxy[3169]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:54 proxmox pveproxy[3170]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:54 proxmox pveproxy[3170]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:54 proxmox pveproxy[3169]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:54 proxmox pveproxy[3169]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:57 proxmox pveproxy[3170]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:57 proxmox pveproxy[3169]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:22:57 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:22:57 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:00 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:00 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:01 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:01 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:04 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:04 proxmox pveproxy[3171]: EV: error in callback (ignoring): Can't call method "push_write" on an undefined value at /usr/share/perl5/PVE/HTTPServer.pm line 295.
Jun 21 00:23:05 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus
Jun 21 00:23:05 proxmox pveproxy[3171]: problem with client 172.20.50.55; rsa_eay_public_decrypt: data too large for modulus


EDIT 2

OK i am pist **f right now... so i will change my plan : if i backup my vm conf, storage.cfg, etc... then i do a pve reinstall by aptitude, then put the conf back in pve, can it solve the problem ?
Click to expand...


I had the same problem two days ago, reinstall pve-manager solved this issue.

Good luck.;)
 
This was a useful tip. I have had a bunch of issues with a 2 node cluster that I was able to resolve with this tip.

Here is how I did it:

  • Separated the cluster by removing the cluster config. I had backed up /etc/pve before I started working on this. The last copy command puts back the conf files for the containers, and you will see them display correctly using a vzlist command after removing the configs.

Code: 
service cman stop
service pve-cluster stop
cp -R /etc/pve/openvz /root/pve_backup
rm /etc/cluster/cluster.conf
rm /var/lib/pve-cluster/*
service pve-cluster start
service cman start 
cp /root/pve_backup/openvz/* /etc/pve/openvz

  • After doing that on both nodes they are effectively separated and /etc/pve is writeable. In my case I also recopied the storage.cfg file as well to restore the old storage configuration
  • Then I tested fully my network and connectivity with the instructions here http://pve.proxmox.com/wiki/Troubleshooting_multicast,_quorum_and_cluster_issues. Its important to do all the steps as simple as they may seem. These clusters need massaging at times and these tests help with that process.
  • Then before building the cluster, I tested each node individually by logging into each individual web interface to see if the node was alive and displaying properly. This was not the first rebuild attempt I have been fighting with this for cluster a while now. There were similar log errors in /var/log/daemon.log to those listed above.
  • One node was fine but on the other I could not access the web interface. I tried different browsers with no luck, clearing browser cache, pvecm updatecerts etc.
  • Tried to restart pvedaemon, pveproxy with no luck
  • Then I stumbled on this post :D and did the cowboy 'apt-get install pve-manager --reinstall' and that immediately did the trick. I used chromium as FF didn't work for me. This was the key point in troubleshooting the issue!!
  • What this gives is a separated cluster and puts the nodes back independently. You can see and manipulate them in their respective web interfaces
  • Then I did the standard 'pvecm create mycluster' on node1 and on node2 'pvecm add ipofnode1'
  • The cluster is green now with both web interfaces showing everything perfectly

I will see how this holds up but so far so good. Some of my problems were enhanced due to LACP and jumbo frames (I reduced back to 1500 MTU and its giving better performance in my case). One more thing to note is that my backup configuration is not there anymore and will need to be recreated.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back