Proxmox 2.0 Cluster: Node Offline, Asking for Password

[Chris Rivera]

1. cluster nodes drop offline
2. node 8 asks for password.

Cluster nodes drop offline:

When this happens the web interface for proxmox shows a few nodes or only node 1 as being online. After restarting the cman and pve-cluster service this seems to clear its self up. Mind you i have to run these commands on node 1 and any other nodes that still show offline after running this command. after 5-10 minutes sometimes they show offline again.... this is aggrevating

Node 8 asks for a password:

All nodes use the same password and are on the same cluster. I can migrate vms to this node but end up being useless once on the node since i cannot access the node. The linux root password for the server is not working. What should i do?

 

[Chris Rivera]

This is the exact error that i recieve when trying to access node 6 and the prompts for the username and password shows up

permission denied - invalid ticket (401)

 

[Chris Rivera]

Complete cluster is offline.

Cannot restart cman service

Stopping cluster:
Stopping dlm_controld... [FAILED]

if i restart pve-cluster nothing happens.



Seemed to be running fine after i got the cluster up and running... now i think i need to reboot the master node causing down time to try to recover from this.

Since all nodes are down there is not possibility to migrate or manage the vms.... can anyone help or guide me to something that can help me?

 

[Chris Rivera]

I did not reboot the master node. But everything today is working perfectly. Must of cleared its self up.


I still have the issue with node 8 where i need a password. I tried using the root password and its not working. For this node if i want to do anything i need to do it via command line?

What do i do to solve this? Can i remove it from the cluster and add it back?

Reinstall Proxmox and add it to the cluster as a new node?

I am trying to use this support forum but don't get any response at all from Staff or Users... i'm guessing this is to push clients to order support?

 

[dietmar]

What do i do to solve this? Can i remove it from the cluster and add it back?

Does that node have correct time? Any hint in /var/log/syslog?

 

[Chris Rivera]

Thanks for the follow up.

A) I was able to update the time on all the nodes but i still have 1 node that show time differently.

I used tzselect:

2 - america
1 - easterntime

= America/New_York

when i run date i get EDT but 1 node still shows HADT. Any way i can fix this or do i need to reboot for the change to take? I already exited the ssh connection and connected and still see the same.

B) So far so good. the only on going issue i still have is cluster nodes dropping offline. Not sure why this is happening. I am using Datacenter quality network devices. I have to use a puppet program to manage the cloud to run the following commands to get the nodes back online:

service cman restart
service pve-cluster restart


Is there any solution to this or a reason why this is happening? Do i need to install an extra program to restart these services? I am up to 8 nodes and wish to keep growing this but the more nodes i add to the cluster the more time it'll take me to manage and restart services 2-3 times a day.


Thank

 

[dietmar]

Is there any solution to this or a reason why this is happening?

Hard to say without additional info. Would you mind to send the syslogs (/var/log/syslog*) for further analysis (directly to dietmar@proxmox.com)?

 

[Chris Rivera]

Just to be clear you want the output of /var/log/syslog*

from all nodes? master node1, or any?

you want me to export the full log as a txt file and email it?


i can cat /var/log/syslog* and theres is a large output.


Just want to be clear before i send you something thats useless and wastes both of our time. thanks for the reply.

 

[dietmar]

from all nodes? master node1, or any?

from one node which went offline.

you want me to export the full log as a txt file and email it?

# tar czf syslog.tgz /var/log/syslog*

Just want to be clear before i send you something thats useless and wastes both of our time. thanks for the reply.

If that is bigger that 7MB we need to find another way, email will not work for such big files here.