proxmox4.2 4 node cluster 1 node went bad

[yaboc]

hi i have 4 node cluster px1-4. due to network power loss the cluster was split

on px1 the faulty one i see

@px1:/etc/pve# pvecm status
Quorum information
------------------
Date: Fri Jun 24 12:12:25 2016
Quorum provider: corosync_votequorum
Nodes: 1
Node ID: 0x00000001
Ring ID: 3024
Quorate: No

Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 1
Quorum: 3 Activity blocked
Flags:

Membership information
----------------------
Nodeid Votes Name
0x00000001 1 10.18.66.30 (local)

the other three px2-4

px4:/etc/pve# pvecm status
Quorum information
------------------
Date: Fri Jun 24 12:12:46 2016
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 0x00000004
Ring ID: 5524
Quorate: Yes

Votequorum information
----------------------
Expected votes: 4
Highest expected: 4
Total votes: 3
Quorum: 3
Flags: Quorate

Membership information
----------------------
Nodeid Votes Name
0x00000002 1 10.18.66.31
0x00000003 1 10.18.66.32
0x00000004 1 10.18.66.33 (local)

so somehow px1 was removed

when i try to add it back

px1:/etc/pve# pvecm add px2
can't create shared ssh key database '/etc/pve/priv/authorized_keys'
authentication key already exists

is there any way to readd it without doing a reinstall and just use the cluster files from the other nodes px2-4 ? If not what's the best way to readd px1 (same ip/hostname)

Thanks

 

[dietmar]

There is no need to re-add the node, because the node is already in the cluster configuration. Instead, try to find out why network communication fails.

 

[yaboc]

dietmar, all nodes can ping between each other now but the gui shows

http://imgur.com/xGNBFOS

and i can't access px1 gui via https://10.18.66.30:8006

i can ping the node from LAN pc and tried to restart pveproxy service

 

[jeffwadsworth]

What error gets logged when you try to restart pveproxy on px1? Make sure these daemons are running on it also.
https://pve.proxmox.com/wiki/Service_daemons

 

[yaboc]

jeffwadsworth

pveproxy restarts and this is what i get in syslog

http://pastebin.com/fqqUrS4T

also all the processes seems to be running

 

[dietmar]

looks like a network/multicast problem...

 

[yaboc]

all nodes have the same hardware and worked fine before, also all are connected to the same switch. plus the px1 can ping everything.

px1:/etc/pve# service cman start
Failed to start cman.service: Unit cman.service failed to load: No such file or directory.

 

[yaboc]

just an update. i decided to reinstall from scratch and re-added the node

 

[jeffwadsworth]

That was the only suggestion I could find from scouring the other forums on this problem, but I didn't find it very helpful advice.