nodes are red but quorum???

[felipe]

hi

tonight a node left and joined the cluster...
now in the gui i see all the others nodes in red and cant work with it...

pvecm nodes
Node Sts Inc Joined Name
1 M 632 2013-12-02 22:06:06 node1
2 M 624 2013-12-02 22:05:40 node7
3 M 632 2013-12-02 22:06:06 node2
4 M 644 2014-02-07 03:31:37 node5
5 M 632 2013-12-02 22:06:06 node4
6 M 632 2013-12-02 22:06:06 node3
7 M 632 2013-12-02 22:06:06 node6
8 M 632 2013-12-02 22:06:06 node8

pvecm status
Version: 6.2.0
Config Version: 8
Cluster Name: cluster01
Cluster Id: 53601
Cluster Member: Yes
Cluster Generation: 644
Membership state: Cluster-Member
Nodes: 8
Expected votes: 8
Total votes: 8
Node votes: 1
Quorum: 5
Active subsystems: 5
Flags:
Ports Bound: 0
Node name: node7
Node ID: 2
Multicast addresses: 239.192.209.51
Node addresses: 192.168.11.7

this looks like the cluster is ok???
but why are the nodes red?

best regards

 

[m.ardito]

what is the output of clustat?
do you see tha same from other nodes (cli and gui)?

Marco

 

[felipe]

all nodes give me the same thing (clustat, pvecm status) and in the gui i only see the local node...

clustat
Cluster Status for cluster01 @ Fri Feb 7 11:11:16 2014
Member Status: Quorate


Member Name ID Status
------ ---- ---- ------
node1 1 Online
node7 2 Online, Local
node2 3 Online
node5 4 Online
node4 5 Online
node3 6 Online
node6 7 Online
node8 8 Online

 

[m.ardito]

network works normally, switches unchanged?

you could try restarting services like:

sudo service pvedaemon restart
sudo service pvestatd restart
sudo service pveproxy restart

Marco

 

[felipe]

nothing changed.
networks seems to work normally
just think that one node left quorum because of heavy load with backup jobs... (2 nodes)

pvedaemon restart
unable to create socket - IO::Socket::INET: Address already in use

 

[felipe]

does anybody have ideas what todo? without rebooting the hosts!!!

 

[felipe]

for the record:
/etc/init.d/pve-cluster restart
/etc/init.d/pvestatd restart
worked.

but mayby somebody of the proxmox team could explain how this can happen? i could not find anything in the logs...

 

[dietmar]

but mayby somebody of the proxmox team could explain how this can happen?

Is there a way to reproduce that behavior?

 

[felipe]

maybe next weekend i will plug und unplug the cable of one node... to try what happens when it leafes and joines the cluster. the strange thing was that everything said it is fine. and the logs had nothing strange- only that this night one node left and joined the cluster...

 

[dietmar]

the strange thing was that everything said it is fine. and the logs had nothing strange- only that this night one node left and joined the cluster...

This usually indicates a problem with 'pvestatd', so restarting this daemon should solve the problem.